will upcoming virtual view feature slice a compound type?

Hdf-forum archives
1

I have a question about the virtual view feature that the hdf5 group is developing. My understanding is that I can build a view dataset out of elements from different datasets of the same type. I'm wondering if I can also build a view dataset that has a different but compatible type. For instance, if I had

datasetA
  type = Compound { float x, int a}

could I make a view dataset that is just of float, that pulls out the float x in the compound from in datasetA?

best,

David

2

Hi David,

I have a question about the virtual view feature that the hdf5 group is developing. My understanding is that I can build a view dataset out of elements from different datasets of the same type. I'm wondering if I can also build a view dataset that has a different but compatible type. For instance, if I had

datasetA
type = Compound { float x, int a}

could I make a view dataset that is just of float, that pulls out the float x in the compound from in datasetA?

HDF5 virtual dataset uses only spacial information for setting a view (or defining mappings), i.e., there is no way to define the self-describing mapping as above. Also, the source datasets have to have compatible datatypes (one type can be converted to another by the HDF5 library). There is no restriction that they have the same datatype.

In HDF5, atomic type and a field of a compound type of the same atomic type are not compatible. This is a current limitation and it can be probably addressed in the future.

Said this, it is possible to pull data out as you described ( we still need to test it!), but it is clunky.

If all source datasets in the mapping use compatible compound datatypes, the library should be able to read or write virtual dataset data by a field. It will be similar to reading a field from a compound dataset.

If the source datasets have non-compatible datatypes, then one can do I/O only on a selection in the virtual dataset that maps to the elements of compatible datatypes. It will be application’s responsibility to make sure that all elements in the selection have compatible types, otherwise I/O will fail - not fun :wink:

Elena

···

On Aug 13, 2015, at 12:30 AM, Schneider, David A. <davidsch@slac.stanford.edu> wrote:

best,

David

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

3

Thanks Elena. The application I had in mind was generating an alternate schema. A data acquisition system may produce a more complicated schema with a number of compound types, while users may need a simpler schema for their analysis. I was wondering if one could generate a virtual view conforming to the simpler schema that gets the data from the more complicated schema. Being able to slice compounds would help, but I think it is not that important. The main thing is to be able to generate a view to the 'big data' which is usually large arrays of float or int's. At least for our DAQ, the data in the compounds tend to be small and I think it would be cheap to make a copy of what users need for their schemas.

best,

David

···

On 08/14/15 14:36, Elena Pourmal wrote:

Hi David,

On Aug 13, 2015, at 12:30 AM, Schneider, David A. <davidsch@slac.stanford.edu> wrote:

I have a question about the virtual view feature that the hdf5 group is developing. My understanding is that I can build a view dataset out of elements from different datasets of the same type. I'm wondering if I can also build a view dataset that has a different but compatible type. For instance, if I had

datasetA
  type = Compound { float x, int a}

could I make a view dataset that is just of float, that pulls out the float x in the compound from in datasetA?

HDF5 virtual dataset uses only spacial information for setting a view (or defining mappings), i.e., there is no way to define the self-describing mapping as above. Also, the source datasets have to have compatible datatypes (one type can be converted to another by the HDF5 library). There is no restriction that they have the same datatype.

In HDF5, atomic type and a field of a compound type of the same atomic type are not compatible. This is a current limitation and it can be probably addressed in the future.

Said this, it is possible to pull data out as you described ( we still need to test it!), but it is clunky.

If all source datasets in the mapping use compatible compound datatypes, the library should be able to read or write virtual dataset data by a field. It will be similar to reading a field from a compound dataset.

If the source datasets have non-compatible datatypes, then one can do I/O only on a selection in the virtual dataset that maps to the elements of compatible datatypes. It will be application�s responsibility to make sure that all elements in the selection have compatible types, otherwise I/O will fail - not fun :wink:

Elena

best,

David

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5