VFD and Direct Chunk API

1

Hello,

It’s not clear to me where the VFD layer stands regarding the direct chunk API. Can I use H5Dread/write_chunk() while using a VFD (such as the GDS one, in which case the const void * buf would be interpreted as device address) ? If I apply some filters on GPU (compression), can I use GDS to write compressed/chunked datasets?

Thanks

2

Hi @samuel.debionne,

yes, the direct chunk API can be used with any VFD since the mechanism of action for the API calls is essentially just to:

  1. Ensure that file space for the chunk is allocated
  2. Retrieve the file address for the chunk
  3. Make an I/O call to the appropriate file address

However, when the library makes that I/O call it has to pass through a layer introduced by the page buffering feature. This feature is disabled by default and so shouldn’t cause any issues, but when enabled the library may attempt to perform memcpy operations on the given buffer, which will of course be problematic in the case of GPU-allocated buffers. The library is not currently intelligent enough to avoid trying to perform memcpy or other similar operations on GPU-allocated buffers in all cases, so there may be other edge cases you run into, but as far as I can tell using the GDS VFD with the direct chunk I/O APIs should work, as long as:

  • The default chunked dataset allocation time of incremental is used without being changed (note that this is only the case for serial HDF5 and not parallel HDF5)
  • The page buffering feature isn’t enabled (it is disabled by default)

The library also shouldn’t care if you applied some filters to the data since it’s just going to pass the buffer directly down to the VFD.

This is also likely the best way to approach the problem currently since the library’s dataset chunk cache causes issues when trying to use the GDS VFD with filters and without the direct chunk API due to memcpy and similar being used elsewhere. Since this hasn’t been tested I can’t give a 100% “yes this will work” answer but if you happen to try this and run into issues, it may help to track down places in the library where we need to be more intelligent about how data buffers are handled internally.

2 Likes