Is there any way to recover possibly corrupted SWMR files?

[furutaka.kazuyoshi]

@nfortne2

The only file operations that are currently allowed in SWMR mode are raw data writes and H5Dset_extent(). H5Fstart_swmr_write() should flush the file. Are you creating all the file metadata prior to calling H5Fstart_swmr_write()?

I checked how I did to prepare an output file and two datasets in it:

1. H5open_f() to Initialize HDF5,
2. H5Get_libversion_f() to get HDF5 version,
3. H5Pcreate_f() to create a file-creation property list,
4. H5Pcreate_f() again to create file-access property list,
5. H5Pset_libver_bounds_f() to limit the version to the latest,
6. H5Fcreate_f( … IOR(H5F_ACC_TRUNC_F,INT(H5F_ACC_SWMR_WRITE_F)) …)) to open an HDF5 file in SWMR mode,
7. H5Screate_simple_f() to create a DataSpace for a simple fixed-length data,
8. H5Dcreate_f() to create a dataset in the data space just created,
9. H5Dwrite_f() to write a data in it,
10. H5Sclose_f() to close the DataSpace, and then
11. H5Dclose_f() to close the DataSet.
12. H5Screate_simple_f to create another DataSpace with half-unlimited dimension,
13. H5Pcreate_f() to create a dataset creation property, then
14. H5Pset_chunk_f()enable chunking and
15. H5Pset_deflate_f() deflate,
16. H5Dcreate_f() create the second DataSet,
17. H5Sclose_f() close the second DataSpace.
18. H5Pclose_f() 3 times to close property lists for dataspace creation, file creation, and file access.
19. H5Fstart_SWMR_write_f() only if the HDF5 version is less than 2.

I did the last step, 19, because I got an error with the call using ver.2.0.0.

I used ver.2.0.0 of the library, both for the calculation which resulted in corrupted files and the one with normal files. The same executable was used. The same batch queue was used, and both were done with OpenMP+MPI hybrid calculations. The only difference between the calculations is the lengths of calculations: a long calculation exceeded the time limit of batch queue and the job was killed; the output HDF5 files were corrupted. Then I did a shorter run with the same parameter; the files were not corrupted.

Previously I did some long calculations with OpenMP only. They were also killed halfway because of time limits of the batch queue, but the resultant files were not corrupted, and I thought it is safe even if the calculation job was killed thanks to SWMR…

I’d like to ask again… What is the proper and recommended way to prepare SWMR writing?

As Dr. @vchoi suggested, is it essential to

creating the file in non-SWMR mode, create the datasets/groups and then call H5Fstart_swmr_write()

Thanks.