# Issue unlocking HDF5 file?

**URL:** <https://forum.hdfgroup.org/t/issue-unlocking-hdf5-file/8552>\
**Category:** HDF5 Library\
**Created:** [June 3, 2021, 5:28pm UTC](https://forum.hdfgroup.org/t/issue-unlocking-hdf5-file/8552 "2021-06-03T17:28:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![robert.bruntz](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/85e7bf/32.png) [@robert.bruntz](https://forum.hdfgroup.org/u/robert.bruntz)\
**Post date:** [June 3, 2021, 5:28pm UTC](https://forum.hdfgroup.org/t/issue-unlocking-hdf5-file/8552/1 "2021-06-03T17:28:55Z")

</div>

We’ve run into a strange, intermittent problem with some HDF5 files.

A large number of files are created while processing data. When interacting with those files much later, a small fraction of those files (\< 10%) cannot be operated on by the current version of h5 tools (h5ls, h5stat, h5dump, etc.). Strangely, this only happens with the original files; a copy of the file works fine. Also, only the current version of the h5 tools, such as h5ls 1.10.6, has trouble; h5ls 1.8.12 (and related) works fine. The original file and its copy are identical in every way that I can find (md5sum, stat, ls -ls, ls -laZ), except for the inode, modification timestamp, and filename - but even the filename can be changed, and the v1.10 h5 tools still don’t work. Also, lslocks and fuser turn up no locks on the problematic file.

Some digging indicated that it’s probably an issue with an HDF5 lock on the file, implemented I think in HDF5 1.10.x. Disabling file locking with an environment variable, such as `HDF5_USE_FILE_LOCKING=FALSE h5ls [badfile].h5`, works for that command, but it doesn’t fix the files. I haven’t been able to figure out where the HDF5 lock is stored (in the superblock? I don’t know where that is stored or how to easily read it), but I did find that `h5clear` should be able to clear the lock (e.g., page 10 here: [https://support.hdfgroup.org/HDF5/docNewFeatures/SWMR/Design-HDF5-FileLocking.pdf](https://support.hdfgroup.org/HDF5/docNewFeatures/SWMR/Design-HDF5-FileLocking.pdf) ), however running `h5clear -s [badfile].h5` returns the terse `h5clear error: h5tools_fopen`, and googling doesn’t turn up anything useful there. Digging into the problematic commands a little more, such as `h5stat --enable-error-stack [badfile].h5`, returns lines like these:  
“#000: H5F.c line 509 in H5Fopen(): unable to open file”  
“#001: H5Fint.c line 1567 in H5F\_open(): unable to lock the file”  
“#002: H5FD.c line 1640 in H5FD\_lock(): driver lock request failed”  
"#003: H5FDsec2.c line 959 in H5FD\_sec2\_lock(): unable to lock file, errno = 11, error message = ‘Resource temporarily unavailable’ "  
which seem like they might be helpful to the right person, but that person is not me.

Does anyone have any insight as to what is going on here?

An ideal solution would be a 1-line command to identify locked files (beyond just trying to read a file and checking the exit code) and (more importantly) a 1-line command to remove a lock that should have already been removed. (But we’ll take any hints that someone might have.)

---

<div class="post-metadata">

**Author:** ![gheber](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/gheber/32/1919_2.png) [@gheber](https://forum.hdfgroup.org/u/gheber)\
**Post date:** [June 8, 2021, 4:07pm UTC](https://forum.hdfgroup.org/t/issue-unlocking-hdf5-file/8552/2 "2021-06-08T16:07:09Z")

</div>

In versions 2 and 3 of the [file format spec.](https://portal.hdfgroup.org/display/HDF5/File+Format+Specification), there is a file consistency flag. Assuming there is no user block, this would be byte 12 (1-based) in the file. What does

```auto
od -j 11 -N 1 <HDF5 file>

```

return?

G.

---

<div class="post-metadata">

**Author:** ![hdf2](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/h/90ced4/32.png) [@hdf2](https://forum.hdfgroup.org/u/hdf2)\
**Post date:** [June 9, 2021, 11:08pm UTC](https://forum.hdfgroup.org/t/issue-unlocking-hdf5-file/8552/3 "2021-06-09T23:08:08Z")

</div>

We had the same issue, but it turned out to be a forked process that kept a file description open. It showed up in `lslocks` as “…” since the file had been renamed since the lock was taken. It could only be seen by comparing the inode numbers of `ls -i` with `cat /proc/locks`.

---

<div class="post-metadata">

**Author:** ![robert.bruntz](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/85e7bf/32.png) [@robert.bruntz](https://forum.hdfgroup.org/u/robert.bruntz)\
**Post date:** [June 12, 2021, 3:23pm UTC](https://forum.hdfgroup.org/t/issue-unlocking-hdf5-file/8552/4 "2021-06-12T15:23:51Z")

</div>

Following up on your suggestion, I checked out the format spec. (BTW - thanks for the link! That answered several questions I had, including where the superblock is.)

Here is the output from `xxd <HDF5 file>`:  
0000000: 8948 4446 0d0a 1a0a 0000 0000 0008 0800 .HDF…

As you can see, byte 12 does not have any of its bits set, so there does not seem to be a lock on the file. This is consistent with the result that a bitwise-identical copy of the file (verified with `md5sum`) _is_ readable by `h5ls` et al.

On the other hand, the fact that the file is readable by h5 tools version 1.8.12 but not 1.10.6, unless the `HDF5_USE_FILE_LOCKING='FALSE'` environment variable is set before the 1.10.6 tool is used, strongly implies that the issue _is_ an HDF5 lock on the file.

One new piece of information that might be relevant: the filesystem that the file is on is mounted over NFS, and I have been told that HDF5 file locks and NFS mounts are a bad combination. (Not my choice - I inherited this issue.) Even then, though, we can’t figure out what mechanism is locking the file.

---

<div class="post-metadata">

**Author:** ![thomas1](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/thomas1/32/1733_2.png) [@thomas1](https://forum.hdfgroup.org/u/thomas1)\
**Post date:** [June 17, 2021, 11:59am UTC](https://forum.hdfgroup.org/t/issue-unlocking-hdf5-file/8552/5 "2021-06-17T11:59:19Z")

</div>

The primary locking is done by the operating system, via the `flock()` function, which doesn’t modify the contents of the file. This fits with what you see - e.g. a copy of the file is unlocked, but renaming it doesn’t unlock it, because the OS knows which inode the lock is associated with. The locks are ‘advisory’, so they only affect processes which try to get the lock, which is why you can bypass them with the `HDF5_USE_FILE_LOCKING` environment variable.

I don’t know why tools like `lslocks` and `fuser` aren’t showing the locks. It wouldn’t surprise me if NFS was part of the answer, though.