VFD SWMR beta 1 release

HDF5 Library
1

Hi all,

We are pleased to announce the Beta 1 pre-release of the HDF5 library with support for the new Single-Writer/Multiple Reader (SWMR) capability, called “VFD SWMR.” VFD SWMR Beta 1 can be obtained from this Github repo:

https://github.com/HDFGroup/hdf5/tree/feature/vfd_swmr_beta_1

VFD SWMR is designed to be a more flexible, more modular, better-performing replacement for the existing SWMR feature.

  • VFD SWMR allows HDF5 objects (groups, datasets, attributes) to be created and destroyed in the course of a reader-writer session. Creating and destroying objects is not possible using the existing SWMR feature.
  • It compartmentalizes much of the SWMR functionality in the bottom layers of the HDF5 library, which both eases the HDF Group’s software maintenance burden, and allows non-SWMR applications to avoid SWMR related overhead.
  • And it makes guarantees for the maximum time from write to availability of data for read, provided that the reading and writing systems and their interconnections can keep up with the data flow.

The VFD SWMR feature has not changed significantly since the prior Alpha 2 release as we’ve been focused on integration and ensuring that the VFD SWMR branch builds and passes tests on a wide variety of platforms and compilers. We’d especially like to get feedback on any platforms where the software cannot build and pass make check / CTest.

Documentation. The doc directory includes

  • VFD_SWMR_RFC_200916.pdf - overview, design, implementation details.
  • vfd-swmr-user-guide.md - quick start guide, sample programs, demos, developer tips.
    You can also find an overview and demo from the recent European HDF User Group entitled Update on new HDF5 VFDs: SWMR, Mirror, Onion.

Known issues in this release:

  • The mirror VFD tests will not compile due to changes in the “zoo” test functionality.
  • OS X 1.10.11 (circa 2015) and earlier do not have clock_gettime(3) and will fail to compile. OS X 1.10.12 and newer should be fine.
  • The Java tests fail on Windows due to internal changes to how H5Fis_accessible() works.
  • Windows, in general, is poorly tested. It will compile and pass all non-Java tests, but we are currently bypassing two tests that have file permission problems. This may just be a test issue and not a fundamental VFD SWMR issue.
  • On Windows, Visual Studio 2015 or newer is required to build the branch.

If you come across any other build or test errors, please let us know so that we can look into the issue.

Please direct these and any other comments, questions, or suggestions to the HDF forum with the words “VFD SWMR” in the subject line.

Regards,

The VFD SWMR Team

2 Likes
2

Hi,

I have one question regarding SWMR.
When you say that several processes can read concurrently the HDF5 file, are the reads really parallel-concurrent? Or is the library serializing the access?

Thank you.

3

SWMR involves independent processes that do communicate via IPC so there is no serialization in the library.

4

Very cool feature that will help us a lot for live data processing.
Is the VFD SWMR going to be part of the HDF5 1.13 release?

5

Clemens, one of the new features that we will release with HDF5 1.13 is the ability to dynamically load VFDs. In other words, you will be able to use VFDs in your application that are not baked into the instance of the HDF5 library you’re running. I believe, all newer VFDs in the pipeline (GPUDirect, Onion, SWMR, etc.) are of that ilk. So the answer is perhaps, “No and yes.” :thinking:
No, in the sense, that there is no longer a strict connection between VFDs and a particular library release (aside from VFL API conformance). Yes, in the sense, that you’ll be able to use VFD SWMR with HDF5 1.13. :champagne:

Best, G.

6

thanks a lot for that explanation, that clears up a lot of things. I wasn’t aware that VFD SWMR is a plugin.

7

VFD SWMR is not a plugin and requires special support from the HDF5 library so it will almost certainly remain an integral part of the library instead of something we dynamically load.

VFD SWMR is still currently in development and won’t make it into HDF5 1.13.0, but it should be released in HDF5 1.13.1, due out sometime next year.