We are pleased to announce the Beta 2 pre-release of the HDF5 library with support for the new Single-Writer/Multiple Reader (SWMR) capability, called “VFD SWMR.” VFD SWMR Beta 2 can be obtained from this Github repo: https://github.com/HDFGroup/hdf5/tree/feature/vfd_swmr_beta_2
VFD SWMR is designed to be a more flexible, more modular, better-performing replacement for the existing SWMR feature.
- VFD SWMR allows HDF5 objects (groups, datasets, attributes) to be created and destroyed in the course of a reader-writer session. Creating and destroying objects is not possible using the existing SWMR feature.
- It compartmentalizes much of the SWMR functionality in the bottom layers of the HDF5 library, which both eases the HDF Group’s software maintenance burden, and allows non-SWMR applications to avoid SWMR related overhead.
- And it makes guarantees for the maximum time from write to availability of data for read, provided that the reading and writing systems and their interconnections can keep up with the data flow.
The main functional difference between the Beta 2 and Beta 1 versions is that Beta 2 supports Virtual Data Sets. Beta 2 also includes performance improvements, bug fixes, and code improvements. We’d especially like to get feedback on any platforms where the software cannot build and pass make check / CTest.
Documentation. The doc directory includes
- VFD_SWMR_RFC_220519.pdf - overview, design, implementation details.
- vfd-swmr-user-guide.md - quick start guide, sample programs, demos, developer tips.
You can also find an overview and demo from the 2021 European HDF User Group entitled Update on new HDF5 VFDs: SWMR, Mirror, Onion on Youtube: https://www.youtube.com/watch?v=QDoKuJojkGI
Known issues and cautions involving his release:
- Don’t run VFD SWMR simultaneously the version of SWMR that was introduced in version 1.10 of the library (“Legacy SWMR”).
- Don’t simultaneously open a file with a regular open and VFD SWMR open.
- The VFD SWMR acceptance tests can take a long time to run and require at least 100 GB of disk space.
- OS X 1.10.11 (circa 2015) and earlier do not have clock_gettime(3) and will fail to compile. OS X 1.10.12 and newer should be fine.
- VFD SWMR is not tested on Windows. On Windows, Visual Studio 2015 or newer is required to build the branch.
If you come across any other build or test errors, please let us know so that we can look into the issue.
Please direct these and any other comments, questions, or suggestions either to the email vfdswmr@hdfgroup.org or as a post to the HDF forum with the words “VFD SWMR” in the subject line.
Regards,
The VFD SWMR Team