# Is there any way to recover possibly corrupted SWMR files? **URL:** \ **Category:** HDF5 Library\ **Created:** [December 22, 2025, 4:33am UTC](https://forum.hdfgroup.org/t/is-there-any-way-to-recover-possibly-corrupted-swmr-files/13631 "2025-12-22T04:33:38Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![furutaka.kazuyoshi](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/f/e5b9ba/32.png) [@furutaka.kazuyoshi](https://forum.hdfgroup.org/u/furutaka.kazuyoshi)\ **Post date:** [December 22, 2025, 4:33am UTC](https://forum.hdfgroup.org/t/is-there-any-way-to-recover-possibly-corrupted-swmr-files/13631/1 "2025-12-22T04:33:38Z")
Dear experts… I submitted a job to a PBS-professional managed queue on our supercomputer. The job is a Monte-Carlo calculation done using MPI and OpenMP and the obtained data are stored with HDF-2.0.0 library (SWMR): no inter-process communication between MPI PEs and therefore sequential HDF5 library is used, and each PE writes its data to its own file. Unfortunately, the job was killed unfinished because it didn’t finish within the allowed time to the queue. All the resultant files cannot be read using e.g. `h5ls`. `h5dump —enable-error-stack=2` to one of the file, `20251217.h5.2`, yields the following message. ```auto HDF5-DIAG: Error detected in HDF5 (2.0.0) thread 1: #000: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5F.c line 821 in H5Fopen(): unable to synchronously open file major: File accessibility minor: Unable to open file #001: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5F.c line 782 in H5F__open_api_common(): unable to open file major: File accessibility minor: Unable to open file #002: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5VLcallback.c line 3869 in H5VL_file_open(): open failed major: Virtual Object Layer minor: Can’t open object #003: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5VLcallback.c line 3718 in H5VL__file_open(): open failed major: Virtual Object Layer minor: Can’t open object #004: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5VLnative_file.c line 128 in H5VL__native_file_open(): unable to open file major: File accessibility minor: Unable to open file #005: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5Fint.c line 2238 in H5F_open(): file is already open for write (may use to clear file consistency flags) major: File accessibility minor: Unable to open file h5dump error: unable to open file “20251217.h5.2” ``` These files have finite sizes, e.g. ```auto -rw-r–r–. 1 furutaka furutaka 12026722 Dec 20 15:29 20251217.h5.2 ``` I did `h5clear -s` to one of the files: after that, doing `h5ls` to the file yields ` **NOT FOUND** `. Is there any way to recover these files? Thanks in advance. Kazuyoshi ---
**Author:** ![nfortne2](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/nfortne2/32/1543_2.png) [@nfortne2](https://forum.hdfgroup.org/u/nfortne2)\ **Post date:** [December 23, 2025, 10:47pm UTC](https://forum.hdfgroup.org/t/is-there-any-way-to-recover-possibly-corrupted-swmr-files/13631/2 "2025-12-23T22:47:14Z")
What happens if you use `h5dump –enable-error-stack=2` on the file after `h5clear -s`? ---
**Author:** ![furutaka.kazuyoshi](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/f/e5b9ba/32.png) [@furutaka.kazuyoshi](https://forum.hdfgroup.org/u/furutaka.kazuyoshi)\ **Post date:** [December 24, 2025, 12:43am UTC](https://forum.hdfgroup.org/t/is-there-any-way-to-recover-possibly-corrupted-swmr-files/13631/3 "2025-12-24T00:43:23Z")
Thanks for the response. > [@nfortne2](#): > > What happens if you use `h5dump –enable-error-stack=2` on the file after `h5clear -s`? as follows… > HDF5-DIAG: Error detected in HDF5 (2.0.0): > #000: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5O.c line 1154 in H5Oget\_info\_by\_name3(): can’t synchronously retrieve object info > major: Object header > minor: Can’t get value > #001: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5O.c line 1129 in H5O\_\_get\_info\_by\_name\_api\_common(): can’t get data model info for object > major: Object header > minor: Can’t get value > #002: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5VLcallback.c line 6088 in H5VL\_object\_get(): get failed > major: Virtual Object Layer > minor: Can’t get value > #003: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5VLcallback.c line 6056 in H5VL\_\_object\_get(): get failed > major: Virtual Object Layer > minor: Can’t get value > #004: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5VLnative\_object.c line 269 in H5VL\_\_native\_object\_get(): object not found > major: Object header > minor: Object not found > #005: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5Gloc.c line 755 in H5G\_loc\_info(): can’t find object > major: Symbol table > minor: Object not found > #006: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5Gtraverse.c line 846 in H5G\_traverse(): internal path traversal failed > major: Symbol table > minor: Object not found > #007: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5Gtraverse.c line 766 in H5G\_\_traverse\_real(): traversal operator failed > major: Symbol table > minor: Can’t move to next iterator location > #008: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5Gloc.c line 716 in H5G\_\_loc\_info\_cb(): can’t get object info > major: Symbol table > minor: Can’t get value > #009: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5Oint.c line 2119 in H5O\_get\_info(): unable to load object header > major: Object header > minor: Unable to protect metadata > #010: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5Oint.c line 1016 in H5O\_protect(): unable to load object header > major: Object header > minor: Unable to protect metadata > #011: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5AC.c line 1303 in H5AC\_protect(): H5C\_protect() failed > major: Object cache > minor: Unable to protect metadata > #012: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5Centry.c line 3154 in H5C\_protect(): can’t load entry > major: Object cache > minor: Unable to load metadata into cache > #013: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5Centry.c line 1227 in H5C\_\_load\_entry(): incorrect metadata checksum after all read attempts > major: Object cache > minor: Read failed > #014: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5Ocache.c line 185 in H5O\_\_cache\_get\_final\_load\_size(): can’t deserialize object header prefix > major: Object header > minor: Unable to decode value > #015: /home/furutaka/work/HDF5/hdf5-2.0.0/src/H5Ocache.c line 1101 in H5O\_\_prefix\_deserialize(): bad object header version number > major: Object header > minor: Wrong version number > h5dump error: internal error (file /home/furutaka/work/HDF5/hdf5-2.0.0/tools/src/h5dump/h5dump.c:line 1548) By the way, the beginning of the file (left after the killed job) before `h5clear -s` looked like ![20251217_SGI_before_h5clear-s](https://dbkt.hdfgroup.org/original/2X/b/b9c5047db60a7a6b86aa0c8ebd31a68bcfbaa335.png) and then after `h5clear -s`, ![20251217_SGI_after_h5clear-s](https://dbkt.hdfgroup.org/original/2X/1/1c680b10614636bf25ed1c564656be60c079553c.png) For reference, in the middle of a Monte-Carlo calculation (another file, sorry): ![20251217_2_SGI_inTheMiddle](https://dbkt.hdfgroup.org/original/2X/c/c9bb085cac11785e97871bdc72ed36da8e5637c1.png) and the one after the job finished properly (i.e. the job was not killed in the middle) : ![20251217_2_SGI](https://dbkt.hdfgroup.org/original/2X/6/6ba571c71f71e620ecd6d0b59e3e2f0425f6be5f.png) As seen in the last image, I made two dataset at the root level, one named `regions` (a simple 1-D array), and `T-eDepEv` (2-D, semi-infinite array). The former is a simple and fixed one and is written before starting the Monte-Carlo calculation. The latter stores event-by-event data in the calculation and will be expanded during the calculation. It seems to me that there’s no superblocks and headers in the failed files (and files in the middle of calculation) (and therefore no way to salvage the files 😢 ). But… When are these information written? p.s. the information on [this page](https://support.hdfgroup.org/documentation/hdf5/latest/_f_m_t_d_i_s_c.html) seems a bit outdated (and contains “`>` in the examples). ---
**Author:** ![nfortne2](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/nfortne2/32/1543_2.png) [@nfortne2](https://forum.hdfgroup.org/u/nfortne2)\ **Post date:** [December 29, 2025, 7:07pm UTC](https://forum.hdfgroup.org/t/is-there-any-way-to-recover-possibly-corrupted-swmr-files/13631/4 "2025-12-29T19:07:45Z")
That doesn’t look like an issue with the superblock, but with an object header (possibly for a dataset). Normally object headers shouldn’t be modified once SWMR write mode is enabled. The way I would approach this is to first find the address of the problematic object header and then look at it in a hex editor to figure out what’s wrong with it. It looks like, from the stack, that either the checksum or the object header version are incorrect, or both (it’s possible the whole thing is zeroes or garbage). If it cannot be recovered, and it is the object header of the dataset you’re looking for, you may be able to find the chunk index by searching for the signature in the file. By default (without adjusting version bounds), this will be a v1 b-tree with signature “TREE”, and you can then use this tree (if it is the right one) to find the chunks with your data. To find the object header, you could use the low level h5debug tool, a custom program that prints out link info before trying to follow the link, or debugging/printf statements in the library. h5debug may also be useful if are unable to fix the object header but you are able to find the b-tree. You may then be able to repair the object header by creating a similar file separately and copying the bytes for the dataset object header to the original file, adjusting the index address and ohdr checksum as necessary. If you can exactly recreate the file (without raw data) the index address may be identical. In this case all you’ll need to do is get the address from the recreated file and copy the object header to the old file. ---
**Author:** ![nfortne2](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/nfortne2/32/1543_2.png) [@nfortne2](https://forum.hdfgroup.org/u/nfortne2)\ **Post date:** [December 30, 2025, 4:53pm UTC](https://forum.hdfgroup.org/t/is-there-any-way-to-recover-possibly-corrupted-swmr-files/13631/5 "2025-12-30T16:53:06Z")
Looking at the hex dumps, it looks like nothing but the superblock was actually flushed to the file. The only file operations that are currently allowed in SWMR mode are raw data writes and H5Dset\_extent(). H5Fstart\_swmr\_write() should flush the file. Are you creating all the file metadata prior to calling H5Fstart\_swmr\_write()? ---
**Author:** ![furutaka.kazuyoshi](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/f/e5b9ba/32.png) [@furutaka.kazuyoshi](https://forum.hdfgroup.org/u/furutaka.kazuyoshi)\ **Post date:** [April 3, 2026, 7:17am UTC](https://forum.hdfgroup.org/t/is-there-any-way-to-recover-possibly-corrupted-swmr-files/13631/6 "2026-04-03T07:17:42Z")
Hi all, long time no see… By constantly consulting [the HDF5 File Format Specification](https://support.hdfgroup.org/documentation/hdf5/latest/_f_m_t4.html) and by comparing the contents of the corrupted files to the normal ones, I finally recovered the corrupted ones. The first 1280 bytes of a corrupted file looks like this: ![corrupted_1st1280bytes](https://dbkt.hdfgroup.org/original/2X/6/6bdfaaad14c8f8fe6981638af172109b4a492f58.png) On the other hand a normal one as follows… ![normal_1st1280bytes](https://dbkt.hdfgroup.org/original/2X/9/902230afc20f72fa3a0cc3bf6d2421c972c4fc91.png) That is, - the superblock is corrupted, - file-consistency flag indicates the file is open for (SWMR) write access, - the EOF address is wrong, - the object header of the root group is missing (which should contain links to two data objects), - one of the object header of the two data objects is missing, whereas the one for the other is properly written. Therefore, I wrote a Python script to - clear the file-consistency flag, - write correct EOF address, and - calculate and write a proper check-sum value for the corrected superblock data, and then - copy the ofject headers for the root group and for the missing data object, and now I can `h5ls`, `h5dump`, and `h5debug` the corrupted files! After this I’ll try to look up the reason for the corruption: though using the same Fortran source (and the same executable), sometimes the files are corrupted but the other times not… By the way, are the outputs of `h5ls` and `h5debug` too terse? (for a corrupted file`h5dump` only says “cannot open file”, `h5ls` “unable to open file”…) This would be because `H5Fopen`only returns `H5I_INVALID_HID` when failed. I’d be happy if `h5debug` spits messages “_useful information for debugging_“ like the following : - the file is now open for writing, (and for SWMR), - the EOF address is too small, - the check-sum for the superblock is wrong, - the root-group header is missing, and so on… Thanks. ---
**Author:** ![gheber](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/gheber/32/1919_2.png) [@gheber](https://forum.hdfgroup.org/u/gheber)\ **Post date:** [April 3, 2026, 10:59am UTC](https://forum.hdfgroup.org/t/is-there-any-way-to-recover-possibly-corrupted-swmr-files/13631/7 "2026-04-03T10:59:10Z")
> [@furutaka.kazuyoshi](#): > > By the way, are the outputs of `h5ls` and `h5debug` too terse? (for a corrupted file`h5dump` only says “cannot open file”, `h5ls` “unable to open file”…) You can get more information by passing the `--enable-error-stack` option. However, that will tell you only which function(s) failed, but not necessarily take you straight to the root cause or the bigger issue. Kudos on your Python script and engaging with the file format specification. I believe that with @vchoi’s [HDF5 pickles](https://github.com/HDFGroup/hdf5-pickles), we are about to open the door to a new breed of tools, including repair tools, that will be helpful before the next “wave” of robustness improvements rolls in. G. ---
**Author:** ![furutaka.kazuyoshi](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/f/e5b9ba/32.png) [@furutaka.kazuyoshi](https://forum.hdfgroup.org/u/furutaka.kazuyoshi)\ **Post date:** [April 6, 2026, 4:47am UTC](https://forum.hdfgroup.org/t/is-there-any-way-to-recover-possibly-corrupted-swmr-files/13631/8 "2026-04-06T04:47:27Z")
> [@nfortne2](#): > > The only file operations that are currently allowed in SWMR mode are raw data writes and H5Dset\_extent(). H5Fstart\_swmr\_write() should flush the file. Are you creating all the file metadata prior to calling H5Fstart\_swmr\_write()? I checked how I did to prepare an output file and two datasets in it: 1. `H5open_f()` to Initialize HDF5, 2. `H5Get_libversion_f()` to get HDF5 version, 3. `H5Pcreate_f()` to create a file-creation property list, 4. `H5Pcreate_f()` again to create file-access property list, 5. `H5Pset_libver_bounds_f()` to limit the version to the latest, 6. `H5Fcreate_f( … IOR(H5F_ACC_TRUNC_F,INT(H5F_ACC_SWMR_WRITE_F)) …))` to open an HDF5 file in SWMR mode, 7. `H5Screate_simple_f()` to create a DataSpace for a simple fixed-length data, 8. `H5Dcreate_f()` to create a dataset in the data space just created, 9. `H5Dwrite_f()` to write a data in it, 10. `H5Sclose_f()` to close the DataSpace, and then 11. `H5Dclose_f()` to close the DataSet. 12. `H5Screate_simple_f` to create another DataSpace with half-unlimited dimension, 13. `H5Pcreate_f()` to create a dataset creation property, then 14. `H5Pset_chunk_f()`enable chunking and 15. `H5Pset_deflate_f()` deflate, 16. `H5Dcreate_f()` create the second DataSet, 17. `H5Sclose_f()` close the second DataSpace. 18. `H5Pclose_f()` 3 times to close property lists for dataspace creation, file creation, and file access. 19. `H5Fstart_SWMR_write_f()` only if the HDF5 version is less than 2. I did the last step, 19, because [I got an error with the call using ver.2.0.0.](https://forum.hdfgroup.org/t/swmr-start-sequence-in-ver-2-0-0-fortran/13609) I used ver.2.0.0 of the library, both for the calculation which resulted in corrupted files and the one with normal files. The same executable was used. The same batch queue was used, and both were done with OpenMP+MPI hybrid calculations. The only difference between the calculations is the lengths of calculations: a long calculation exceeded the time limit of batch queue and the job was killed; the output HDF5 files were corrupted. Then I did a shorter run with the same parameter; the files were not corrupted. Previously I did some long calculations with OpenMP only. They were also killed halfway because of time limits of the batch queue, but the resultant files were not corrupted, and I thought it is safe even if the calculation job was killed thanks to SWMR… I’d like to ask again… What is the proper and recommended way to prepare SWMR writing? As [Dr. @vchoi suggested](https://forum.hdfgroup.org/t/swmr-start-sequence-in-ver-2-0-0-fortran/13609/4), is it essential to > creating the file in non-SWMR mode, create the datasets/groups and then call H5Fstart\_swmr\_write() Thanks. ---
**Author:** ![furutaka.kazuyoshi](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/f/e5b9ba/32.png) [@furutaka.kazuyoshi](https://forum.hdfgroup.org/u/furutaka.kazuyoshi)\ **Post date:** [April 6, 2026, 4:49am UTC](https://forum.hdfgroup.org/t/is-there-any-way-to-recover-possibly-corrupted-swmr-files/13631/9 "2026-04-06T04:49:47Z")
> [@gheber](#): > > Kudos on your Python script and engaging with the file format specification. I believe that with @vchoi’s [HDF5 pickles](https://github.com/HDFGroup/hdf5-pickles), we are about to open the door to a new breed of tools, including repair tools, that will be helpful before the next “wave” of robustness improvements rolls in. Thanks for the info! I’ll try [it](https://github.com/HDFGroup/hdf5-pickles). Kazuyoshi ---
**Author:** ![vchoi](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/v/a9adbd/32.png) [@vchoi](https://forum.hdfgroup.org/u/vchoi)\ **Post date:** [April 6, 2026, 4:44pm UTC](https://forum.hdfgroup.org/t/is-there-any-way-to-recover-possibly-corrupted-swmr-files/13631/10 "2026-04-06T16:44:34Z")
The API will fail if the file is already opened/created with SWMR-write mode.