VFD SWMR beta 1 release


#1

Hi all,

We are pleased to announce the Beta 1 pre-release of the HDF5 library with support for the new Single-Writer/Multiple Reader (SWMR) capability, called “VFD SWMR.” VFD SWMR Beta 1 can be obtained from this Github repo:

https://github.com/HDFGroup/hdf5/tree/feature/vfd_swmr_beta_1

VFD SWMR is designed to be a more flexible, more modular, better-performing replacement for the existing SWMR feature.

  • VFD SWMR allows HDF5 objects (groups, datasets, attributes) to be created and destroyed in the course of a reader-writer session. Creating and destroying objects is not possible using the existing SWMR feature.
  • It compartmentalizes much of the SWMR functionality in the bottom layers of the HDF5 library, which both eases the HDF Group’s software maintenance burden, and allows non-SWMR applications to avoid SWMR related overhead.
  • And it makes guarantees for the maximum time from write to availability of data for read, provided that the reading and writing systems and their interconnections can keep up with the data flow.

The VFD SWMR feature has not changed significantly since the prior Alpha 2 release as we’ve been focused on integration and ensuring that the VFD SWMR branch builds and passes tests on a wide variety of platforms and compilers. We’d especially like to get feedback on any platforms where the software cannot build and pass make check / CTest.

Documentation. The doc directory includes

  • VFD_SWMR_RFC_200916.pdf - overview, design, implementation details.
  • vfd-swmr-user-guide.md - quick start guide, sample programs, demos, developer tips.
    You can also find an overview and demo from the recent European HDF User Group entitled Update on new HDF5 VFDs: SWMR, Mirror, Onion.

Known issues in this release:

  • The mirror VFD tests will not compile due to changes in the “zoo” test functionality.
  • OS X 1.10.11 (circa 2015) and earlier do not have clock_gettime(3) and will fail to compile. OS X 1.10.12 and newer should be fine.
  • The Java tests fail on Windows due to internal changes to how H5Fis_accessible() works.
  • Windows, in general, is poorly tested. It will compile and pass all non-Java tests, but we are currently bypassing two tests that have file permission problems. This may just be a test issue and not a fundamental VFD SWMR issue.
  • On Windows, Visual Studio 2015 or newer is required to build the branch.

If you come across any other build or test errors, please let us know so that we can look into the issue.

Please direct these and any other comments, questions, or suggestions to the HDF forum with the words “VFD SWMR” in the subject line.

Regards,

The VFD SWMR Team


#2

Hi,

I have one question regarding SWMR.
When you say that several processes can read concurrently the HDF5 file, are the reads really parallel-concurrent? Or is the library serializing the access?

Thank you.


#3

SWMR involves independent processes that do communicate via IPC so there is no serialization in the library.