# VOL Datatype Handling **URL:** \ **Category:** Hdf-forum archives\ **Created:** [April 22, 2016, 9:03am UTC](https://forum.hdfgroup.org/t/vol-datatype-handling/3743 "2016-04-22T09:03:49Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![juliankunkel](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/j/a9adbd/32.png) [@juliankunkel](https://forum.hdfgroup.org/u/juliankunkel)\ **Post date:** [April 22, 2016, 9:03am UTC](https://forum.hdfgroup.org/t/vol-datatype-handling/3743/1 "2016-04-22T09:03:49Z")
Dear all, I was trying to provide an alternative implementation in VOL to deal with the datatypes. For testing purpose, it will simply keep the data structures in memory. In particular the functions are problematic: - VOL\_datatype\_open() - VOL\_datatype\_get() Internally, the H5T\_construct\_datatype() function calls datatype\_get to fetch the serialized form. This requires the datatype to be serialized in a particular form. In the Dev guide I was reading "8. The Named Datatype Pickle" > **[main.pdf](https://svn.hdfgroup.org/hdf5doc/trunk/RFCs/HDF5/VOL/developer_guide/main.pdf)** > > 152.14 KB I do not completely understand the rationale to differentiate between datatype\_open and datatype\_commit and relying on the serialized form: Since VOL\_datatype\_commit() receives the user-space datatype as "hid\_t", I would have expected that datatype\_open returns a hid to a proper created datatype. i.e., i could internally create the datatype with end user functions. This would give me the opportunity to store it internally in any format that suits me. In a primitive in-memory implementation, I believe the following should work: the datatype\_commit() function could store a copy of the datatype in memory, then return again this copy when datype\_open() is called. Looking forward to the discussion. Thanks & regards, Julian > **路路路** > > -- > [http://wr.informatik.uni-hamburg.de/people/julian\_kunkel](http://wr.informatik.uni-hamburg.de/people/julian_kunkel) ---
**Author:** ![Mohamad\_Chaarawi](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/m/e47c2d/32.png) [@Mohamad\_Chaarawi](https://forum.hdfgroup.org/u/Mohamad_Chaarawi)\ **Post date:** [May 2, 2016, 7:17pm UTC](https://forum.hdfgroup.org/t/vol-datatype-handling/3743/2 "2016-05-02T19:17:25Z")
Hi Julian, Apologies for the late reply.. > Dear all, > I was trying to provide an alternative implementation in VOL to deal > with the datatypes. HDF5 Datatypes are in-memory objects that are implemented in HDF5. The VOL plugin class for datatypes is just to deal with Named/Committed datatypes that are stored in the HDF5 file, but is in no way designed to change how the Library implements and uses the datatypes themselves. The VOL class is just to provide a different way to store the name datatype in the file. So I don't understand here what you are trying to accomplish when you say to provide an alternative implementation in the VOL to deal with datatypes.. If it it's just for storing the datatype in the file, then we can move on to your next questions, otherwise, you need to stop here and realize that the VOL is for nothing more than this 馃檪 > For testing purpose, it will simply keep the data structures in memory. > > In particular the functions are problematic: > - VOL\_datatype\_open() > - VOL\_datatype\_get() > > Internally, the H5T\_construct\_datatype() function calls datatype\_get > to fetch the serialized form. > This requires the datatype to be serialized in a particular form. In > the Dev guide I was reading "8. The Named Datatype Pickle" > [https://svn.hdfgroup.org/hdf5doc/trunk/RFCs/HDF5/VOL/developer\_guide/main.pdf](https://svn.hdfgroup.org/hdf5doc/trunk/RFCs/HDF5/VOL/developer_guide/main.pdf) > > I do not completely understand the rationale to differentiate between > datatype\_open and datatype\_commit and relying on the serialized form: > Since VOL\_datatype\_commit() receives the user-space datatype as "hid\_t", > I would have expected that datatype\_open returns a hid to a proper > created datatype. > i.e., i could internally create the datatype with end user functions. > This would give me the opportunity to store it internally in any > format that suits me. > > In a primitive in-memory implementation, I believe the following > should work: the datatype\_commit() function could store a copy of the > datatype in memory, then return again this copy when datype\_open() is > called. I still don't understand your use case here, but I will try to explain the situation here again as I did in the developer guide that you pointed to. In order for the library to function properly with datatypes, it will always need the NATIVE implementation of HDF5 datatypes, not any other implementation. Again, the VOL only gives you the flexibility to store the native HDF5 datatype in whatever way you want. You could serialize it and store in a binary blob, or whatever.. BUT the Library will always need the native (internal) HDF5 H5T\_t struct for datatypes to function properly in all the pieces of code for datasets, attributes, etc... In order to do that, the VOL plugin must preserve a native representation for the native datatype which can be accomplished by serializing and deserializing the datatype struct using H5Tencode/decode. Again, I don't know if I understood what you are aiming to do here, or if I addressed your comments, but there is so much I can do to explain this using email, since this is a pretty complex issue in the first place. Thanks, Mohamad > **路路路** > > On 4/22/16, 4:03 AM, "Hdf-forum on behalf of Julian Kunkel" \ wrote: > > > Looking forward to the discussion. > > > > Thanks & regards, > > Julian > > > > -- > > [http://wr.informatik.uni-hamburg.de/people/julian\_kunkel](http://wr.informatik.uni-hamburg.de/people/julian_kunkel) > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@lists.hdfgroup.org > > [http://lists.hdfgroup.org/mailman/listinfo/hdf-forum\_lists.hdfgroup.org](http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org) > > Twitter: [https://twitter.com/hdf5](https://twitter.com/hdf5) ---
**Author:** ![juliankunkel](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/j/a9adbd/32.png) [@juliankunkel](https://forum.hdfgroup.org/u/juliankunkel)\ **Post date:** [May 3, 2016, 10:48am UTC](https://forum.hdfgroup.org/t/vol-datatype-handling/3743/3 "2016-05-03T10:48:19Z")
Dear Mohamad, thanks for your reply; Sorry for my suboptimal expression: Absolutely, the use case you describe is what I have in mind. I was thinking to store the committed datatypes in different way, in fact potentially in a database and not any file. But for the first implementation I wanted to store data only in memory (for performance testing and backwards compatibility). > I still don't understand your use case here, but I will try to explain the situation here again as I did in the developer guide that you pointed to. > In order for the library to function properly with datatypes, it will always need the NATIVE implementation of HDF5 datatypes, not any other implementation. > Again, the VOL only gives you the flexibility to store the native HDF5 datatype in whatever way you want. You could serialize it and store in a binary blob, or whatever.. BUT the Library will always need the native (internal) HDF5 H5T\_t struct for datatypes to function properly in all the pieces of code for datasets, attributes, etc... Yes, that is absolutely clear and not what I try to accomplish (see below the obstacle). > In order to do that, the VOL plugin must preserve a native representation for the native datatype which can be accomplished by serializing and deserializing the datatype struct using H5Tencode/decode. Looking at the current implementation in the native plugin: H5VL\_native\_datatype\_get() calls H5T\_encode() which is defined as a private function and should not be used by any other VOL implementation and it is expected to return. I propose to move out this logic and return a completely constructed datatype as "hid\_t". Since VOL\_type\_commit() also receives such a type it feels natural that datatype\_get() returns a userspace datatype as hid\_t as well. Since apparently nobody else is interested in this detailed discussion, we can continue the discussion offline from the list. Regards, Julian > **路路路** > > -- > [http://wr.informatik.uni-hamburg.de/people/julian\_kunkel](http://wr.informatik.uni-hamburg.de/people/julian_kunkel) ---
**Author:** ![Mohamad\_Chaarawi](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/m/e47c2d/32.png) [@Mohamad\_Chaarawi](https://forum.hdfgroup.org/u/Mohamad_Chaarawi)\ **Post date:** [May 3, 2016, 1:36pm UTC](https://forum.hdfgroup.org/t/vol-datatype-handling/3743/4 "2016-05-03T13:36:17Z")
Hi Julian, > Looking at the current implementation in the native plugin: > H5VL\_native\_datatype\_get() calls H5T\_encode() which is defined as a > private function and should not be used by any other VOL > implementation and it is expected to return. Right, but you can use the public version of this function: [https://www.hdfgroup.org/HDF5/doc/RM/RM\_H5T.html#Datatype-Encode](https://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-Encode) And the decode: [https://www.hdfgroup.org/HDF5/doc/RM/RM\_H5T.html#Datatype-Decode](https://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-Decode) > I propose to move out this logic and return a completely constructed > datatype as "hid\_t". We can't return an hid\_t directly from the plugin to the user, as the HDF5 library (the higher level VL layer) takes whatever structure returned from the plugins for objects and wraps it with another structure to include some more information about the plugin this object is being created with and a reference count on that object, and then creates the hid\_t on that higher level struct. So we can't have plugin do all of that. > Since VOL\_type\_commit() also receives such a type it feels natural > that datatype\_get() returns a userspace datatype as hid\_t as well. > Since apparently nobody else is interested in this detailed > discussion, we can continue the discussion offline from the list. Sure I'm happy to continue this discussion offline, and maybe schedule a quick call to explain the quirks with this 馃檪 Thanks, Mohamad > **路路路** > > > Regards, > > Julian > > > > -- > > [http://wr.informatik.uni-hamburg.de/people/julian\_kunkel](http://wr.informatik.uni-hamburg.de/people/julian_kunkel) > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@lists.hdfgroup.org > > [http://lists.hdfgroup.org/mailman/listinfo/hdf-forum\_lists.hdfgroup.org](http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org) > > Twitter: [x.com](https://twitter.com/hdf5)