Cannot append to a file with SWMR once opened (and closed...)

HDF5 Library h5py
1

To assist reproducing bugs, please include the following:

  • Operating System: CentOS 8
  • Where Python was acquired: Anaconda
  • The full traceback/stack trace shown (if it appears)
  • Python, h5py and HDF5 version: see below

h5py.version.info contains the needed versions, which can be displayed by

python -c 'import h5py; print(h5py.version.info)'

>>> import h5py
>>> print(h5py.version.info)
Summary of the h5py configuration
---------------------------------

h5py    3.9.0
HDF5    1.12.2
Python  3.11.4 (main, Jul  5 2023, 14:15:25) [GCC 11.2.0]
sys.platform    linux
sys.maxsize     9223372036854775807
numpy   1.25.0
cython (built with) 0.29.35
numpy (built against) 1.23.2
HDF5 (built against) 1.12.2

where python should be substituted for the path to python used to install
h5py with.

I am trying to use the h5py version with the swmr feature.
After a read access with swmr=True as argument, the following error shows up
[Errno 11] Unable to open file (unable to lock file, errno = 11, error message = ‘Resource temporarily unavailable’)

And there are no way to continue to append to that file, even without closing the file. When i open for append, i specify hf.swmr_mode=True, as detailed in the documentation.

If might be useful, this is the function for appending, while for reading, i use something such

import h5py
percorso = f"/path/to/file.h5"
with h5py.File(percorso, 'r', libver='latest', swmr=True) as hf:
     # Do something
    print(hf.keys())

Once I run the previous code, it become IMPOSSIBLE to append to the same file.

Does someone has some ideas or suggestion?

Thanks in advance
Best regards

Stefano

2

It looks like your writer code creates datasets after switching into SWMR mode. This isn’t allowed - you have to set up your datasets first, then switch to SWMR write mode.

If you think this isn’t the issue, please do show the traceback to illustrate which line the error is coming from.

3

Is it correct that even the first time I open the file in append mode?
Or the first time should be opened in write mode?

Anyway, I’ll do soon the test you suggested me. It is quite strange that last year the same code worked…
Thanks for your response.

Stefano

4

Dear Thomas,

Thanks for your help

I am sorry to inform you this didn’t solve the problem
By considering the writer code i moved the line hf.swmr_mode = True inside the else (at line 179)

The error keep being the same
[Errno 11] Unable to open file (unable to lock file, errno = 11, error message = ‘Resource temporarily unavailable’)

The reader code is the same

The main code that calls the function I linked is this one which is basically a loop for calling that function

What am I doing wrong?

5

I’m not sure what the issue is, then. Could you post the traceback? This is the block of lines above the error message that shows what code was running when the error happened.

Opening in append mode ('a') should be fine - if the file doesn’t exist before, it will be created.

6

Here you are

Traceback (most recent call last):
File “/home/scarsi-home/202307_PROVAHDF5/insulabtestbeamv3/data_process_scripts/./HDF5.py”, line 72, in
scriviDati(outfile, infile, settings[numWaveform_key])
File “/home/scarsi-home/202307_PROVAHDF5/insulabtestbeamv3/data_process_scripts/commonFunctions.py”, line 150, in scriviDati
with h5py.File(HDF5file, ‘a’, libver=‘latest’) as hf:
File “/home/scarsi-home/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py”, line 533, in init
fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
File “/home/scarsi-home/anaconda3/lib/python3.9/site-packages/h5py/_hl/files.py”, line 238, in make_fid
fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
File “h5py/_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper
File “h5py/_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper
File “h5py/h5f.pyx”, line 106, in h5py.h5f.open
BlockingIOError: [Errno 11] Unable to open file (unable to lock file, errno = 11, error message = ‘Resource temporarily unavailable’)

7

OK, so this is on opening the file, before you switch it into SWMR mode. The error message means it can’t get the lock on the file. This probably means that another process has the file open. Note that for SWMR, the idea is that your writer process goes into SWMR mode before your readers start and then keeps the file open. It’s not really designed for the write to close and reopen the file.

If you need to get around that, you can set an environment variable to disable file locking - however, read the warning a bit further down that page.

8

The idea i use this file format is that i have an unique data file and when some particle detector acquires some data, producing a different file, let’s say every minute, i append this subset of data to my hdf5 file. My script wait for a new data file and append it. I guarantee i have a single writer. At the same time, many readers can use this file for doing data analysis

Do you think in my case it is worth to disable that environment variable, or do you have any kind of suggestion with respect to my situation?

The strange think is that one year ago I used HDF5 files this way and nothing happened; and by the way i have never read about this intended way of using HDF5 files…

9

If your script stays running waiting for new data to appear, it should be easy enough for it to keep the HDF5 file it’s writing to open.

If that’s not an option, I can’t really say if it’s suitable to disable the file locking. It might be fine, but it might also be fine 99.9% of the time but very occasionally crash, so you should consider how bad it would be if it went wrong.