I have a question about the virtual view feature that the hdf5 group is developing. My understanding is that I can build a view dataset out of elements from different datasets of the same type. I'm wondering if I can also build a view dataset that has a different but compatible type. For instance, if I had
datasetA
type = Compound { float x, int a}
could I make a view dataset that is just of float, that pulls out the float x in the compound from in datasetA?
best,
David
Hi David,
I have a question about the virtual view feature that the hdf5 group is developing. My understanding is that I can build a view dataset out of elements from different datasets of the same type. I'm wondering if I can also build a view dataset that has a different but compatible type. For instance, if I had
datasetA
type = Compound { float x, int a}could I make a view dataset that is just of float, that pulls out the float x in the compound from in datasetA?
HDF5 virtual dataset uses only spacial information for setting a view (or defining mappings), i.e., there is no way to define the self-describing mapping as above. Also, the source datasets have to have compatible datatypes (one type can be converted to another by the HDF5 library). There is no restriction that they have the same datatype.
In HDF5, atomic type and a field of a compound type of the same atomic type are not compatible. This is a current limitation and it can be probably addressed in the future.
Said this, it is possible to pull data out as you described ( we still need to test it!), but it is clunky.
If all source datasets in the mapping use compatible compound datatypes, the library should be able to read or write virtual dataset data by a field. It will be similar to reading a field from a compound dataset.
If the source datasets have non-compatible datatypes, then one can do I/O only on a selection in the virtual dataset that maps to the elements of compatible datatypes. It will be application’s responsibility to make sure that all elements in the selection have compatible types, otherwise I/O will fail - not fun 
Elena
On Aug 13, 2015, at 12:30 AM, Schneider, David A. <davidsch@slac.stanford.edu> wrote:
best,
David
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Thanks Elena. The application I had in mind was generating an alternate schema. A data acquisition system may produce a more complicated schema with a number of compound types, while users may need a simpler schema for their analysis. I was wondering if one could generate a virtual view conforming to the simpler schema that gets the data from the more complicated schema. Being able to slice compounds would help, but I think it is not that important. The main thing is to be able to generate a view to the 'big data' which is usually large arrays of float or int's. At least for our DAQ, the data in the compounds tend to be small and I think it would be cheap to make a copy of what users need for their schemas.
best,
David
On 08/14/15 14:36, Elena Pourmal wrote:
Hi David,
On Aug 13, 2015, at 12:30 AM, Schneider, David A. <davidsch@slac.stanford.edu> wrote:
I have a question about the virtual view feature that the hdf5 group is developing. My understanding is that I can build a view dataset out of elements from different datasets of the same type. I'm wondering if I can also build a view dataset that has a different but compatible type. For instance, if I had
datasetA
type = Compound { float x, int a}could I make a view dataset that is just of float, that pulls out the float x in the compound from in datasetA?
HDF5 virtual dataset uses only spacial information for setting a view (or defining mappings), i.e., there is no way to define the self-describing mapping as above. Also, the source datasets have to have compatible datatypes (one type can be converted to another by the HDF5 library). There is no restriction that they have the same datatype.
In HDF5, atomic type and a field of a compound type of the same atomic type are not compatible. This is a current limitation and it can be probably addressed in the future.
Said this, it is possible to pull data out as you described ( we still need to test it!), but it is clunky.
If all source datasets in the mapping use compatible compound datatypes, the library should be able to read or write virtual dataset data by a field. It will be similar to reading a field from a compound dataset.
If the source datasets have non-compatible datatypes, then one can do I/O only on a selection in the virtual dataset that maps to the elements of compatible datatypes. It will be application�s responsibility to make sure that all elements in the selection have compatible types, otherwise I/O will fail - not fun
Elena
best,
David
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
