HDF5 File locking on ubuntu

HDF5 Library
1

Hi everyone

I’m facing following problem:

  1. My Software writes a HDF5 File with data every five minutes (one hdf5 file per 5 minutes)
  2. In parallel I want to use h5repack on completed files with an delay of 5-10 minutes

On Windows it works without problems. On linux I get the following error while writing to the newest file (lets say file xxxx_105000.h5) while in a parallel process h5repack is started with the file xxxx_104000.h5. So even if h5repack does not use “xxxx_105000.h5” I have problems to lock this file. If I disable file locking by using HDF5_USE_FILE_LOCKING everything works fine. Any idea why this happens and I am not able to write to a newer file while I run h5repack in parallel on a file that was written 10 minutes ago?

Thanks for any ideas!

HDF5-DIAG: Error detected in HDF5 (1.10.9) thread 1:
#000: hdf5-1.10.9/src/H5F.c line 413 in H5Fopen(): unable to open file
major: File accessibility
minor: Unable to open file
#001: hdf5-1.10.9/src/H5Fint.c line 1735 in H5F_open(): unable to lock the file
major: File accessibility
minor: Unable to lock file
#002: hdf5-1.10.9/src/H5FD.c line 1625 in H5FD_lock(): driver lock request failed
major: Virtual File Layer
minor: Unable to lock file
#003:hdf5-1.10.9/src/H5FDsec2.c line 1002 in H5FD__sec2_lock(): unable to lock file, errno = 11, error message = ‘Resource temporarily unavailable’
major: Virtual File Layer
minor: Unable to lock file

2

Technically you are not writing concurrently into the file, but sequentially using some synchronisation primitive, looking at the error message the one you compiled into libhdf5.

On linux advisory locking is a popular choice using flock() , lockf() , and fcntl(). So you have an option of running software 1, once it completes, then software 2; or before entering into the critical section check if the advisory lock is present, and wait until you are clear to go. Then invoke h5repack from a process that checks for the lock similarly. It is common practice to invoke programs using fork+exec.
hope it helps.