meaning of H5S_ALL_F for mem_space_id in H5Dwrite

Hi,

I want to select a hyperslab in a file, but use all of the memory space, so my call to H5Dwrite looks like this (fortran):

call h5dwrite_f(dset_id, dtreal_id, real(mygrid(grid)%u(:,1,lm)), &
                array_dims, error, H5S_ALL_F,dspace_id )

Things to note here:
- I use H5S_ALL_F (otherwise the default) for the memspace selection
- I use dspace_id for the file space selection, which has been populated
  using an earlier call to h5sselect_hyperslab_f.

What I see is that, while the file space selection works fine (the correct hyperslab is selected and written to, the offset of the hyperslab is also applied to the memory space. I didn't expect this, because I specifically pass H5S_ALL_F, which according to the documentation:

  H5S_ALL in C (H5S_ALL_F in FORTRAN) is the default value and indicates that the whole dataspace in memory is selected for the I/O operation.

After not finding an error in my code I dove into the hdf5 source and found this in H5Dio.c:685

    /* Initialize dataspace information */
    if(!file_space)
        file_space = dataset->shared->space;
    if(!mem_space)
        mem_space = file_space;

Note the last line. While if file_space is 0 (H5S_ALL), the entire space of the dataset is used, the same is not true for the memory space: if you happen to set file_space (as I do), but not mem_space (as I do), then mem_space is *not* set to the size of the memory dataspace, but to the (passed) size of the file space. Which means in that case any selection applied to the file dataspace is also applied to the memory data space.

Am I correct that this is a bug in the HDF5 implementation, or is this intended behavior? If it is intended, isn't this then a bug in the documentation?

thanks, Frank

...

Which means in that case any selection applied to the file dataspace is also applied to the memory data space.

Am I correct that this is a bug in the HDF5 implementation, or is this intended behavior? If it is intended, isn't this then a bug in the documentation?

Does anyone know or just want to comment?

Frank

···

On Fri, Jun 09, 2017 at 03:36:29PM -0500, Frank Loeffler wrote:

This behavior is documented in H5Dwrite https://support.hdfgroup.org/HDF5/doc/RM/RM_H5D.html#Dataset-Write

(yep...confusing)

Elena

···

On Jun 14, 2017, at 10:33 AM, Frank Loeffler <knarf@cct.lsu.edu<mailto:knarf@cct.lsu.edu>> wrote:

On Fri, Jun 09, 2017 at 03:36:29PM -0500, Frank Loeffler wrote:
...
Which means in that case any selection applied to the file dataspace is also applied to the memory data space.

Am I correct that this is a bug in the HDF5 implementation, or is this intended behavior? If it is intended, isn't this then a bug in the documentation?

Does anyone know or just want to comment?

Frank

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Elena,

Thanks. I must have missed that. This is indeed confusing. I am currently also looking at the tutorial at

  https://support.hdfgroup.org/HDF5/Tutor/rdwt.html

which states: "Note that H5S_ALL is passed in for the memory and file dataspace parameters in the read and write calls. This indicates that the entire dataspace of the dataset will be read or written to."

While this is technically correct (since in the examples both the memory and the file dataspace are specified as H5S_ALL), this is indeed a confusing statement. Maybe a note could be added that 'H5S_ALL' by itself does not necessarily have that meaning, and the meaning of specifying H5S_ALL for one dataspace indeed depends on the other dataspace.

In general, this is quite a confusing choice of definition for H5S_ALL. Is there another define that does what I actually thought H5S_ALL does:
simply selecting the entire available space, independent of whatever was selected for the other data space? My best guess is: no, since I don't see anything mentioned in H5Spublic.h. It would be a useful addition.

There is of course the simple workaround of specifying a space for both, but requiring a separate data space for a simple array just because the other space has an offset seems a bit awkward.

Frank

···

On Thu, Jun 15, 2017 at 09:21:40PM +0000, Elena Pourmal wrote:

This behavior is documented in H5Dwrite https://support.hdfgroup.org/HDF5/doc/RM/RM_H5D.html#Dataset-Write

(yep...confusing)

Hi Frank,

This behavior is documented in H5Dwrite https://support.hdfgroup.org/HDF5/doc/RM/RM_H5D.html#Dataset-Write

(yep...confusing)

Hi Elena,

Thanks. I must have missed that.

It is hard to find. I had to think little-bit before I recalled where to find this info :wink:

This is indeed confusing. I am currently also looking at the tutorial at

https://support.hdfgroup.org/HDF5/Tutor/rdwt.html

which states: "Note that H5S_ALL is passed in for the memory and file dataspace parameters in the read and write calls. This indicates that the entire dataspace of the dataset will be read or written to."

While this is technically correct (since in the examples both the memory and the file dataspace are specified as H5S_ALL), this is indeed a confusing statement. Maybe a note could be added that 'H5S_ALL' by itself does not necessarily have that meaning, and the meaning of specifying H5S_ALL for one dataspace indeed depends on the other data space.

Thank you for suggestion! We definitely need to have a better explanation how selections and data spaces work. I entered JIRA report HDFFV-10229.

In general, this is quite a confusing choice of definition for H5S_ALL. Is there another define that does what I actually thought H5S_ALL does:
simply selecting the entire available space, independent of whatever was selected for the other data space? My best guess is: no, since I don't see anything mentioned in H5Spublic.h. It would be a useful addition.

Noted.

There is of course the simple workaround of specifying a space for both, but requiring a separate data space for a simple array just because the other space has an offset seems a bit awkward.

Agree.

Thank you for reporting!

Elena

···

On Jun 15, 2017, at 4:39 PM, Frank Loeffler <knarf@cct.lsu.edu> wrote:
On Thu, Jun 15, 2017 at 09:21:40PM +0000, Elena Pourmal wrote:

Frank

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5