We have been using hdf5 1.10.6 in production for a number of years, and with great stability on Windows and Linux. In our production setup we use a writer process that generates hdf files with many realisations of a simulation process within each hdf file. Each realisation (case) is stored within a group in the hdf file. At the same time other processes will access the data (read only). In release 1.10.6 this has been accomplished using the SWMR feature and flushing the data after completion of each realisation (case).
Now we are trying to upgrade to 1.14.6 in order to prepare for e.g. cloud usage and make use of other improvements in the system. But, this does not work out well for us. Not beeing experts on the internals of hdf5, we have, with assistance of ChatGPT been able to build the release with Intel fortran support and zlib compression. The writer produces the hdf files, but when the reader tries to access the data at the same time, an access error is given. When the writer is finished, the reader is able to access the data as in 1.10.6.
we have tried adding h5pset_file_locking_f, h5pset_fclose_degree_f with H5F_CLOSE_WEAK_F and h5pset_libver_bounds_f with H5F_LIBVER_EARLIEST_F, H5F_LIBVER_v110_F but nothing seems to work as we want.
When we use 1.10.6 in the writer and 1.14.6 in the reader, it works fine. I suspect that there may be an issue witht the file locking and/or flushing operations but I am unable to find any documentation to explain these issues to me.
We would appreciate very much if anyone could provide us with suggestions for solving this issue.
I guess the basic question is - is SWMR nor supported in the 1.14 series of hdf5?
To answer your last question, yes, SWMR is supported in 1.14.6. If your SWMR workflow worked in 1.10.6, it should work with 1.14.6. If that is not the case, then it seems there is a bug.
We had a similar issue with fortran writer using 1.10.7 and a java reader using 1.14.4. It worked without any modifications on Linux, but under Windows we needed to disable the locking on the reader.
var accessFlags = HDF5Constants.H5F_ACC_RDONLY | HDF5Constants.H5F_ACC_SWMR_READ;
var accessProps = HDF5Constants.H5P_DEFAULT;
// disable file locking on windows so writer can write to file
// https://support.hdfgroup.org/documentation/hdf5/latest/_file_lock.html
if (Utilities.isWindows())
{
accessProps = H5.H5Pcreate(HDF5Constants.H5P_FILE_ACCESS);
H5.H5Pset_file_locking(accessProps, false, true);
}
fileId = H5.H5Fopen(hdfFile, accessFlags, accessProps);
if (Utilities.isWindows())
{
H5.H5Pclose(accessProps);
}
thanks for the response. I also noted the same issue with 1.10.7, which is the reason we waited until 1.14.6 before trying to upgrade (the 1.12 series being “faulty” as we understood at the time).
a quick followup: I notice that you are modifying the reader. I thought that it was the writer that locked the file so the reader could not access the file?
Now it works. We can view the results with the viewer while the writer runs in 1.14.6.
We cannot view the results of the writer using 1.14.6 with the reader of 1.10.6 but this is not critical since we do not support backward compatibility. Fortunately we can, however, read the results of the writer_1.14.6 with the reader_1.10.6 after completion of the simulations.
In order to make it work we had to use the IOR function with the swmr parameters, e.g.
the IOR operator (ifort compiler) would not accept using the H5F_ACC_SWMR_READ_F as input because it is of a different type than H5F_ACC_RDONLY_F (real versus integer?). Making the new integer parameter (prefixed int_) with the same bits inside as the H5F_ACC_SWMR_READ_F for the reader and H5F_ACC_SWMR_WRITE_F for the writer solved this issue, And then it started to work…
I don’t know if this may be related to the “Windows bug” we started out with. But I describe it here if someone wants to adress the issue.
