Problems with HDF5 and SWMR on network disk when program writing to the HDF5 file and program reading from the HDF5 file is running on the same host.
I’m using HDF5 library version 1.10.4 on RH7. The network disk is mounted with the NSF file system. I have also tested the same C++ code with version 1.12.0 and the issue is still there.
The issue that we experience is a significant hang (10-15 seconds) for the program writing, ones the reading program opens the output file. The writing resumes and at the end the output file is not damaged from this. New data is written to the output file every half second, hence a hang of 10-15 second is significant.
What is interesting, is that If I run the writing program on a different host that the read program (still with the same disk location), I don’t see any hang at all.
I have tried running the exact same binary files (executables and input files) on the same host but now using the /tmp disk. When I do this, I don’t see any hang.
Is this an issue that is known to HDF5 ? Have other reported this ?
From section 2 of HDF5 Single-Writer Multiple-Reader User’s Guide:
This approach is based on the assumption that the HDF5 file resides on a file system compliant with
POSIX write semantics. The ordering feature of the POSIX write semantics is critical for the
implementation: POSIX I/O writes must be performed in a sequentially consistent manner.
These semantics apply to any processes that access the file from any location.
The HDF5 SWMR feature was tested on Linux clusters with GPFS and Lustre, Linux systems with ext3/4,
FreeBSD with UFS2, OS X with HFS+, and Windows with NTFS and GPFS. The SWMR feature does not
work on network file systems (NFS, SMB/Windows file shares, and so on) since write ordering may not
be preserved on those systems.
There is change on the horizon: RFC: VFD SWMR See section 2.1.1.2 NFS.
G.
1 Like
Thank you very much for a thorough and good answer. It was not the answer we had hoped for since hdf5 seemed very promising for our project. Hopefully there will eventually be a solution for NFS as well and then we will evaluate this again.
Have a look at this presentation from yesterday’s European HUG event. The recording of the presentation should pop up on our YouTube channel soon. You can download an alpha version of SWMR VFD and take it for a spin.
Best, G.
