RFC for new Dataspace routines

HDF5 Library
1

Hi All,

Please find attached an RFC that describes a couple of dataspace routines that we plan to add to the HDF5 API in the near future. If you have the time, please give it a read and feel free to send us comments.
We would like to hear from you if you see that you could benefit from those routines but would like to change something or would like us to consider adding other routines. It would be great, in either case, if you could include your use case.

Thank you,
Mohamad

Selection_RFC.docx (49.9 KB)

2

Mohamad

herr_t H5Dtransfer (hid_t src_space_id, const void *src_buf, H5T_t type, hid_t dst_space_id, /*out*/void *dst_buf);

I'm not certain that this will be any use on its own. If a selection is gigabytes and the VOL layer only has a small buffer available, then it needs to be able to make these transfers in pieces, performing puts/sends or whatever as appropriate. The function would be more useful if it was re-entrant and had a buffer size_type so that the selection could be copied by the first N, then next N, then next N until exhausted.

herr_t H5Sselect_iterate (hid_t dataspace_id, H5S_select_iterator_t op, void *op_data);

This function would be great and should be higher priority than the first because you can do the first using this second one - and don't have the problem of the limited buffer size. The user can iterate over the dataspace and copy as much or as little on each entry to the callback function as desired and maintain their own book keeping of where they left off. If the selection has huge contiguous chunks, the user callback can break these into pieces and perform the appropriate copies as substeps. If the selection is very sparse, then an internal buffer can be filled and acted upon as the iterations progress.

the callback function
herr_t (*H5S_select_iterate_t)(hsize_t *offset_coords, hsize_t length, void * op_data);

could be improved by adding a user callback void *pointer so that when you call iterate - you pass a pointer to the function - and also a user pointer to a data structure of the user's choice, which is passed to the callback as a user parameter. This way we can track intermediate transfer objects (like if we only partially transferred data or are filling an internal buffer) and in the case of multiple threads acting on these iterations, we can make sure each thread has its own data pointer an avoid static/global objects which will not be safe.

I just wrote this off the top of my head, so criticism welcome.

JB

···

-----Original Message-----
From: h5vol-bounces@hdfgroup.org [mailto:h5vol-bounces@hdfgroup.org] On Behalf Of Mohamad Chaarawi
Sent: 21 August 2012 23:45
To: HDF5 Virtual Object Layer (VOL) Discussions; hdf-forum@hdfgroup.org; hdf5lib@hdfgroup.org
Subject: [H5vol] RFC for new Dataspace routines

Hi All,

Please find attached an RFC that describes a couple of dataspace routines that we plan to add to the HDF5 API in the near future. If you have the time, please give it a read and feel free to send us comments.

We would like to hear from you if you see that you could benefit from those routines but would like to change something or would like us to consider adding other routines. It would be great, in either case, if you could include your use case.

Thank you,

Mohamad

3

Hi All,

After a good first set of comments, we revised the RFC a bit and decided to add two new routines (for scatter and gather), that should address the "large data that I do not have enough resources to handle" case (thanks to John for pointing that out). Please find attached the revised RFC.
Again more comments are appreciated.

Thanks,
Mohamad

Selection_RFC.docx (54 KB)

···

On 8/21/2012 4:45 PM, Mohamad Chaarawi wrote:

Hi All,

Please find attached an RFC that describes a couple of dataspace routines that we plan to add to the HDF5 API in the near future. If you have the time, please give it a read and feel free to send us comments.
We would like to hear from you if you see that you could benefit from those routines but would like to change something or would like us to consider adding other routines. It would be great, in either case, if you could include your use case.

Thank you,
Mohamad

_______________________________________________
H5vol mailing list
H5vol@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/h5vol_hdfgroup.org

4

Hi John,

Mohamad

**

*herr_t H5Dtransfer (hid_t src_space_id, const void *src_buf, H5T_t type, hid_t dst_space_id, /*out*/void *dst_buf);*

I'm not certain that this will be any use on its own. If a selection is gigabytes and the VOL layer only has a small buffer available, then it needs to be able to make these transfers in pieces, performing puts/sends or whatever as appropriate. The function would be more useful if it was re-entrant and had a buffer size_type so that the selection could be copied by the first N, then next N, then next N until exhausted.

This would be more of what the second routine would do (as you mention later). This routine was originally gather/scatter routines that we have mostly implemented internally and would like to expose for users. We then decided to make it more general.
Granted that if the data is very large compared to memory available, the second routine would be better to break up the requests in case of a client-server plugin.

*herr_t H5Sselect_iterate (hid_t dataspace_id, H5S_select_iterator_t op, void *op_data);*

This function would be great and should be higher priority than the first because you can do the first using this second one - and don't have the problem of the limited buffer size. The user can iterate over the dataspace and copy as much or as little on each entry to the callback function as desired and maintain their own book keeping of where they left off. If the selection has huge contiguous chunks, the user callback can break these into pieces and perform the appropriate copies as substeps. If the selection is very sparse, then an internal buffer can be filled and acted upon as the iterations progress.

The reason that the first is a priority I would say is because that it is very easy to implement (if not already implemented internally) and would fit our needs directly. We are still scoping user needs for the iterate routine and see if it needs to be adjusted.

the callback function

*herr_t (*H5S_select_iterate_t)(hsize_t *offset_coords, hsize_t length, void * op_data);*

could be improved by adding a user callback void *pointer so that when you call iterate -- you pass a pointer to the function - and also a user pointer to a data structure of the user's choice, which is passed to the callback as a user parameter. This way we can track intermediate transfer objects (like if we only partially transferred data or are filling an internal buffer) and in the case of multiple threads acting on these iterations, we can make sure each thread has its own data pointer an avoid static/global objects which will not be safe.

I'm not sure I quite understand what you are suggesting here. the callback (H5S_select_iterator_t op) IS a user callback. The user specifies also his own udata (void *op_data) in the iterate routine, that will get passed to his specified callback too. The select_iterate routine here is similar to, for example, the dataset iterate (H5Diterate) operation that gets called on elements of a dataset, but here it calls on selection coordinates.
If you want another callback, you can add it the the user data (op_data) or just call it from the user callback (op).

Thanks,
Mohamad

···

On 8/22/2012 8:56 AM, Biddiscombe, John A. wrote:

I just wrote this off the top of my head, so criticism welcome.

JB

-----Original Message-----
From: h5vol-bounces@hdfgroup.org [mailto:h5vol-bounces@hdfgroup.org] On Behalf Of Mohamad Chaarawi
Sent: 21 August 2012 23:45
To: HDF5 Virtual Object Layer (VOL) Discussions; hdf-forum@hdfgroup.org; hdf5lib@hdfgroup.org
Subject: [H5vol] RFC for new Dataspace routines

Hi All,

Please find attached an RFC that describes a couple of dataspace routines that we plan to add to the HDF5 API in the near future. If you have the time, please give it a read and feel free to send us comments.

We would like to hear from you if you see that you could benefit from those routines but would like to change something or would like us to consider adding other routines. It would be great, in either case, if you could include your use case.

Thank you,

Mohamad

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

5

Hi John,
  Mohamad and I kicked around another pair of routines that should meet your goals for the iterative scatter/gather routines, and I've tried to describe them below. Let me know what you think.

  Quincey

···

=========================================================================

herr_t H5Dgather(hid_t src_space_id, const void *src_buf, hid_t type, size_t dst_buf_size, void *dst_buf, H5D_gather_func_t op, void *op_data);

typedef herr_t (*H5D_gather_func_t)(const void *dst_buf, size_t dst_buf_bytes_used, void *op_data);

The H5Dgather routine would gather [at most] dst_buf_size bytes from the source buffer, according to the selection in the source dataspace and [common] datatype, into the destination buffer, then call the application's op_data callback, giving the application a chance to "drain" the destination buffer. If more than dst_buf_size bytes worth of data are available in the source selection, H5Dgather would repeatedly call the application's callback routine.

herr_t H5Dscatter(size_t src_buf_size, void *src_buf, hid_t type, hid_t dst_space_id, void *dst_buf, H5D_scatter_func_t op, void *op_data);

typedef herr_t (*H5D_scatter_func_t)(void *src_buf, size_t *src_buf_bytes_used, void *op_data);

Similar to the H5Dgather routine, the H5Dscatter routine would call the application's op_data callback to fill up the source buffer with data (returning the number of bytes used in the source buffer through the src_buf_bytes_used parameter), and scatter those values into the destination buffer, according to the destination selection and the [common] datatype. Repeated calls to the application callback will be made if more than src_buf_size bytes worth of data is needed to fill the destination selection.

On Aug 22, 2012, at 8:56 AM, Biddiscombe, John A. wrote:

Mohamad

herr_t H5Dtransfer (hid_t src_space_id, const void *src_buf, H5T_t type, hid_t dst_space_id, /*out*/void *dst_buf);
I’m not certain that this will be any use on its own. If a selection is gigabytes and the VOL layer only has a small buffer available, then it needs to be able to make these transfers in pieces, performing puts/sends or whatever as appropriate. The function would be more useful if it was re-entrant and had a buffer size_type so that the selection could be copied by the first N, then next N, then next N until exhausted.

herr_t H5Sselect_iterate (hid_t dataspace_id, H5S_select_iterator_t op, void *op_data);
This function would be great and should be higher priority than the first because you can do the first using this second one - and don’t have the problem of the limited buffer size. The user can iterate over the dataspace and copy as much or as little on each entry to the callback function as desired and maintain their own book keeping of where they left off. If the selection has huge contiguous chunks, the user callback can break these into pieces and perform the appropriate copies as substeps. If the selection is very sparse, then an internal buffer can be filled and acted upon as the iterations progress.

the callback function
herr_t (*H5S_select_iterate_t)(hsize_t *offset_coords, hsize_t length, void * op_data);
could be improved by adding a user callback void *pointer so that when you call iterate – you pass a pointer to the function - and also a user pointer to a data structure of the user’s choice, which is passed to the callback as a user parameter. This way we can track intermediate transfer objects (like if we only partially transferred data or are filling an internal buffer) and in the case of multiple threads acting on these iterations, we can make sure each thread has its own data pointer an avoid static/global objects which will not be safe.

I just wrote this off the top of my head, so criticism welcome.

JB

-----Original Message-----
From: h5vol-bounces@hdfgroup.org [mailto:h5vol-bounces@hdfgroup.org] On Behalf Of Mohamad Chaarawi
Sent: 21 August 2012 23:45
To: HDF5 Virtual Object Layer (VOL) Discussions; hdf-forum@hdfgroup.org; hdf5lib@hdfgroup.org
Subject: [H5vol] RFC for new Dataspace routines

Hi All,

Please find attached an RFC that describes a couple of dataspace routines that we plan to add to the HDF5 API in the near future. If you have the time, please give it a read and feel free to send us comments.
We would like to hear from you if you see that you could benefit from those routines but would like to change something or would like us to consider adding other routines. It would be great, in either case, if you could include your use case.

Thank you,
Mohamad

_______________________________________________
Hdf5lib mailing list
Hdf5lib@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf5lib_hdfgroup.org

6

Quincey, Mohamad

The new H5Dgather looks good, but I am concerned about one issue. The fact that we assume a common datatype between buffers. I am accustomed to ignoring datatype issues when working with hdf5 and simply get on with float/double etc and can happily read double into float or vice versa.
One project coming along will be using BlueGene and requires interoperability with another cluster attached which will be x86. Here we will have big/little endian issues and it'd be nice if some of the internals of hdf5 which handle this could be leveraged.

Is there any way we can use the routines for gather scatter and iteration that you have proposed in the last few messages in conjunction with datatype changes such as float/double/ long/int or big/little endian conversion?

JB

···

From: h5vol-bounces@hdfgroup.org [mailto:h5vol-bounces@hdfgroup.org] On Behalf Of Quincey Koziol
Sent: 22 August 2012 20:27
To: hdf5lib@hdfgroup.org
Cc: HDF5 Virtual Object Layer (VOL) Discussions; hdf-forum@hdfgroup.org
Subject: Re: [H5vol] [Hdf5lib] RFC for new Dataspace routines

Hi John,
            Mohamad and I kicked around another pair of routines that should meet your goals for the iterative scatter/gather routines, and I've tried to describe them below. Let me know what you think.

            Quincey

=========================================================================

herr_t H5Dgather(hid_t src_space_id, const void *src_buf, hid_t type, size_t dst_buf_size, void *dst_buf, H5D_gather_func_t op, void *op_data);

typedef herr_t (*H5D_gather_func_t)(const void *dst_buf, size_t dst_buf_bytes_used, void *op_data);

The H5Dgather routine would gather [at most] dst_buf_size bytes from the source buffer, according to the selection in the source dataspace and [common] datatype, into the destination buffer, then call the application's op_data callback, giving the application a chance to "drain" the destination buffer. If more than dst_buf_size bytes worth of data are available in the source selection, H5Dgather would repeatedly call the application's callback routine.

herr_t H5Dscatter(size_t src_buf_size, void *src_buf, hid_t type, hid_t dst_space_id, void *dst_buf, H5D_scatter_func_t op, void *op_data);

typedef herr_t (*H5D_scatter_func_t)(void *src_buf, size_t *src_buf_bytes_used, void *op_data);

Similar to the H5Dgather routine, the H5Dscatter routine would call the application's op_data callback to fill up the source buffer with data (returning the number of bytes used in the source buffer through the src_buf_bytes_used parameter), and scatter those values into the destination buffer, according to the destination selection and the [common] datatype. Repeated calls to the application callback will be made if more than src_buf_size bytes worth of data is needed to fill the destination selection.

On Aug 22, 2012, at 8:56 AM, Biddiscombe, John A. wrote:

Mohamad

herr_t H5Dtransfer (hid_t src_space_id, const void *src_buf, H5T_t type, hid_t dst_space_id, /*out*/void *dst_buf);
I'm not certain that this will be any use on its own. If a selection is gigabytes and the VOL layer only has a small buffer available, then it needs to be able to make these transfers in pieces, performing puts/sends or whatever as appropriate. The function would be more useful if it was re-entrant and had a buffer size_type so that the selection could be copied by the first N, then next N, then next N until exhausted.

herr_t H5Sselect_iterate (hid_t dataspace_id, H5S_select_iterator_t op, void *op_data);
This function would be great and should be higher priority than the first because you can do the first using this second one - and don't have the problem of the limited buffer size. The user can iterate over the dataspace and copy as much or as little on each entry to the callback function as desired and maintain their own book keeping of where they left off. If the selection has huge contiguous chunks, the user callback can break these into pieces and perform the appropriate copies as substeps. If the selection is very sparse, then an internal buffer can be filled and acted upon as the iterations progress.

the callback function
herr_t (*H5S_select_iterate_t)(hsize_t *offset_coords, hsize_t length, void * op_data);
could be improved by adding a user callback void *pointer so that when you call iterate - you pass a pointer to the function - and also a user pointer to a data structure of the user's choice, which is passed to the callback as a user parameter. This way we can track intermediate transfer objects (like if we only partially transferred data or are filling an internal buffer) and in the case of multiple threads acting on these iterations, we can make sure each thread has its own data pointer an avoid static/global objects which will not be safe.

I just wrote this off the top of my head, so criticism welcome.

JB

-----Original Message-----
From: h5vol-bounces@hdfgroup.org<mailto:h5vol-bounces@hdfgroup.org> [mailto:h5vol-bounces@hdfgroup.org]<mailto:[mailto:h5vol-bounces@hdfgroup.org]> On Behalf Of Mohamad Chaarawi
Sent: 21 August 2012 23:45
To: HDF5 Virtual Object Layer (VOL) Discussions; hdf-forum@hdfgroup.org<mailto:hdf-forum@hdfgroup.org>; hdf5lib@hdfgroup.org<mailto:hdf5lib@hdfgroup.org>
Subject: [H5vol] RFC for new Dataspace routines

Hi All,

Please find attached an RFC that describes a couple of dataspace routines that we plan to add to the HDF5 API in the near future. If you have the time, please give it a read and feel free to send us comments.
We would like to hear from you if you see that you could benefit from those routines but would like to change something or would like us to consider adding other routines. It would be great, in either case, if you could include your use case.

Thank you,
Mohamad

_______________________________________________
Hdf5lib mailing list
Hdf5lib@hdfgroup.org<mailto:Hdf5lib@hdfgroup.org>
http://mail.hdfgroup.org/mailman/listinfo/hdf5lib_hdfgroup.org

7

Hi John,
  OK, good, we'll go ahead with the scatter/gather routines (Mohamad, can you add them to the RFC?).

  As far as datatypes are concerned, I think that there's probably support for what you want to do. There's the H5Tconvert() routine for converting a buffer of elements from one datatype to another, if custom datatype conversions are needed, an application can register new conversion routines with H5Tregister(), there's mechanisms for intercepting overflows and other exceptions with H5Pset_type_conv_cb(), datatypes themselves can be serialized and transferred from one system to another with H5Tencode()/H5Tdecode().

  Is there anything you want to do that isn't covered by these routines?

  Quincey

···

On Aug 23, 2012, at 4:23 AM, Biddiscombe, John A. wrote:

Quincey, Mohamad

The new H5Dgather looks good, but I am concerned about one issue. The fact that we assume a common datatype between buffers. I am accustomed to ignoring datatype issues when working with hdf5 and simply get on with float/double etc and can happily read double into float or vice versa.
One project coming along will be using BlueGene and requires interoperability with another cluster attached which will be x86. Here we will have big/little endian issues and it’d be nice if some of the internals of hdf5 which handle this could be leveraged.

Is there any way we can use the routines for gather scatter and iteration that you have proposed in the last few messages in conjunction with datatype changes such as float/double/ long/int or big/little endian conversion?

JB

From: h5vol-bounces@hdfgroup.org [mailto:h5vol-bounces@hdfgroup.org] On Behalf Of Quincey Koziol
Sent: 22 August 2012 20:27
To: hdf5lib@hdfgroup.org
Cc: HDF5 Virtual Object Layer (VOL) Discussions; hdf-forum@hdfgroup.org
Subject: Re: [H5vol] [Hdf5lib] RFC for new Dataspace routines

Hi John,
            Mohamad and I kicked around another pair of routines that should meet your goals for the iterative scatter/gather routines, and I've tried to describe them below. Let me know what you think.

            Quincey

=========================================================================

herr_t H5Dgather(hid_t src_space_id, const void *src_buf, hid_t type, size_t dst_buf_size, void *dst_buf, H5D_gather_func_t op, void *op_data);

typedef herr_t (*H5D_gather_func_t)(const void *dst_buf, size_t dst_buf_bytes_used, void *op_data);

The H5Dgather routine would gather [at most] dst_buf_size bytes from the source buffer, according to the selection in the source dataspace and [common] datatype, into the destination buffer, then call the application's op_data callback, giving the application a chance to "drain" the destination buffer. If more than dst_buf_size bytes worth of data are available in the source selection, H5Dgather would repeatedly call the application's callback routine.

herr_t H5Dscatter(size_t src_buf_size, void *src_buf, hid_t type, hid_t dst_space_id, void *dst_buf, H5D_scatter_func_t op, void *op_data);

typedef herr_t (*H5D_scatter_func_t)(void *src_buf, size_t *src_buf_bytes_used, void *op_data);

Similar to the H5Dgather routine, the H5Dscatter routine would call the application's op_data callback to fill up the source buffer with data (returning the number of bytes used in the source buffer through the src_buf_bytes_used parameter), and scatter those values into the destination buffer, according to the destination selection and the [common] datatype. Repeated calls to the application callback will be made if more than src_buf_size bytes worth of data is needed to fill the destination selection.

On Aug 22, 2012, at 8:56 AM, Biddiscombe, John A. wrote:

Mohamad

herr_t H5Dtransfer (hid_t src_space_id, const void *src_buf, H5T_t type, hid_t dst_space_id, /*out*/void *dst_buf);
I’m not certain that this will be any use on its own. If a selection is gigabytes and the VOL layer only has a small buffer available, then it needs to be able to make these transfers in pieces, performing puts/sends or whatever as appropriate. The function would be more useful if it was re-entrant and had a buffer size_type so that the selection could be copied by the first N, then next N, then next N until exhausted.

herr_t H5Sselect_iterate (hid_t dataspace_id, H5S_select_iterator_t op, void *op_data);
This function would be great and should be higher priority than the first because you can do the first using this second one - and don’t have the problem of the limited buffer size. The user can iterate over the dataspace and copy as much or as little on each entry to the callback function as desired and maintain their own book keeping of where they left off. If the selection has huge contiguous chunks, the user callback can break these into pieces and perform the appropriate copies as substeps. If the selection is very sparse, then an internal buffer can be filled and acted upon as the iterations progress.

the callback function
herr_t (*H5S_select_iterate_t)(hsize_t *offset_coords, hsize_t length, void * op_data);
could be improved by adding a user callback void *pointer so that when you call iterate – you pass a pointer to the function - and also a user pointer to a data structure of the user’s choice, which is passed to the callback as a user parameter. This way we can track intermediate transfer objects (like if we only partially transferred data or are filling an internal buffer) and in the case of multiple threads acting on these iterations, we can make sure each thread has its own data pointer an avoid static/global objects which will not be safe.

I just wrote this off the top of my head, so criticism welcome.

JB

-----Original Message-----
From: h5vol-bounces@hdfgroup.org [mailto:h5vol-bounces@hdfgroup.org] On Behalf Of Mohamad Chaarawi
Sent: 21 August 2012 23:45
To: HDF5 Virtual Object Layer (VOL) Discussions; hdf-forum@hdfgroup.org; hdf5lib@hdfgroup.org
Subject: [H5vol] RFC for new Dataspace routines

Hi All,

Please find attached an RFC that describes a couple of dataspace routines that we plan to add to the HDF5 API in the near future. If you have the time, please give it a read and feel free to send us comments.
We would like to hear from you if you see that you could benefit from those routines but would like to change something or would like us to consider adding other routines. It would be great, in either case, if you could include your use case.

Thank you,
Mohamad

_______________________________________________
Hdf5lib mailing list
Hdf5lib@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf5lib_hdfgroup.org

_______________________________________________
H5vol mailing list
H5vol@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/h5vol_hdfgroup.org