Hi, folks!
The VFD interface provides a good abstraction for hardware-accelerated I/O routines. John Ravi’s GDS VFD is one such example: it uses NVIDIA’s GPUDirect Storage to initiate I/O to a cudaMalloc
'd buffer provided by the caller.
Different VFDs may deal with memory managed by other specialized allocators: a VFD optimized to read from Samsung’s SmartSSD would need to perform I/O to a buffer allocated via OpenCL’s clCreateBuffer()
, for instance.
However, I don’t want to bloat every application that needs to support said VFDs by introducing #ifdefs
and explicit calls to device-specific memory allocators. For this reason I started to look around to find which kind of abstraction the VFD interface provides for memory management.
I have noticed that the H5FD_class_t
provides two members named alloc
and free
that I’ve initially presumed to be in charge of interfacing with specialized memory allocators (which would be just perfect):
-
H5FD__alloc_real(H5FD_t *file, ...)
calls the VFDalloc
handler -
H5FD__free_real(H5FD_t *file, ...)
calls the VFDfree
handler
However, code comments (“allocate space in the file”) and the APIs below suggest that alloc
and free
relate to fallocate()
/ ftruncate()
instead:
-
H5FD_alloc(H5FD_t *file, ..., H5F_t *f, ...)
marks the filef
end-of-allocated flag dirty -
H5FDalloc(H5FD_t *file, ..., H5F_t *f, ...)
modifies the dataset access property list
In the absence of memory-management APIs at the VFD layer I’d like to ask if this is something that’s captured somebody else’s attention before. Is there someone working on this topic already? If that’s not the case then I’d be happy to prototype an extension to H5FD_class_t
as long as that’s useful to the community.
Such an API would be useful for two primary reasons. First, applications could be as vendor/VFD-neutral as possible. Second, I remember having seen some code in the I/O filter path (H5Z*.c
) which assumed that the output buffers had been allocated with malloc
. If that code were to call into the VFD to reallocate buffers that wouldn’t be a problem anymore.
Thanks!
Lucas