RFC: VFD SWMR (virtual file driver single writer/multiple reader)

Hi all,

The VFD SWMR RFC has been posted on our support portal site (and here as well, for your convenience.)

VFD_SWMR_RFC_191027.pdf (1.5 MB)

The purpose of the SWMR (Single Writer Multiple Reader) feature is to allow a second
process to read a HDF5 file while data is being written to it. Use cases range from
monitoring data collection and/or steering experiments in progress to financial
applications.

The existing SWMR implementation touches most parts of the HDF5 library, and
therefore presents significant maintenance issues. Further, it offers no guarantees on
the maximum time from write to read – which makes it problematic for some
applications.

The primary impetus behind VFD SWMR is to implement SWMR in a more modular
fashion – thus minimizing maintenance costs. Fringe benefits include allowing the
HDF5 library to make guarantees for the maximum time from write to availability of
data for read (subject to the performance limits of the underlying file system, and
presuming that the writer calls the HDF5 library frequently), and the possibility of
extending SWMR to NFS and object stores.

This feature was briefly demonstrated during our October 25th 2019 webinar, and can be seen at this point in the webinar recording. We welcome your comments on this here in the forum, or if you prefer to email privately, you can send those to Elena by email at epourmal -at- hdfgroup.org.

Thank you,

The HDF Group