Concurrent read while writing HDF5 file (without SWMR): intermittent metadata / object header errors — any supported workaround?

HDF5 Library
1

Hello

I am trying to read an HDF5 file while it is being written by another process, without using SWMR mode. I understand that this may not be officially supported, but I would like to understand whether there is any safe or semi-safe workaround for my use case, or whether this is fundamentally impossible.

I will describe the setup, what works initially, and the errors I eventually hit.

System / environment

  • Platform: Windows 10 x64

  • Compiler: MSVC (Visual Studio 2022)

  • HDF5 version: 1.14.5 (from Conan)

  • Language: C++ (HDF5 C++ API + some direct C API calls)

  • Storage: local NTFS filesystem

  • Processes: separate writer process and reader process (no MPI)

Use case

  • One process writes time steps incrementally into an HDF5 file.

  • New groups, subgroups, and datasets are created as the simulation progresses.

  • Another process repeatedly opens the file, reads available data, then closes it.

  • The reader does not keep the file handle open permanently.

  • No SWMR mode (this is intentional for now).

Expected behavior (best case):

  • Reader may fail to see the newest data, but should not crash.

  • On retry, reader should eventually see newly added groups/datasets.

What I tried

Reader side

  • File opened read-only (H5F_ACC_RDONLY)

  • Tried:

    • disabling file locking

    • opening a fresh file handle on every retry

    • exponential backoff between retries

  • No cached HDF5 objects kept between iterations

Writer side

  • File opened with H5F_ACC_RDWR or created with H5F_ACC_TRUNC

  • Groups and datasets are created dynamically

  • H5Fflush(..., H5F_SCOPE_GLOBAL) is called after writes

Initially, this sometimes works:

  • Reader can read the file correctly for a few iterations

  • New time steps appear as expected

Errors encountered

After some time, the reader fails with hard HDF5 errors, not just “object not found”.

Example 1 — dataset read failure

H5Dread(): can't synchronously read data
...
H5FD_read(): addr overflow, addr = 3299221668, size = 8, eoa = 2922366596
major: Invalid arguments to routine
minor: Address overflowed

Example 2 — object header / metadata corruption (most worrying)

H5O_get_info(): unable to load object header
H5C__verify_len_eoa(): address of object past end of allocation
major: Object cache
minor: Bad value

At this point the reader crashes consistently even after the writer finishes, and I stay on the same process of the loop to try to read each time.

My understanding so far

From reading documentation and debugging:

  • These errors appear when the reader sees partially written metadata

  • The file’s EOA (end of allocation) changes while the reader is accessing metadata

  • HDF5 correctly rejects inconsistent object headers

My question

I fully understand that SWMR is the supported solution, but before fully restructuring my pipeline, I would like to ask:

  1. Is there any documented or undocumented workaround to make this pattern safer without SWMR?

    • e.g. specific flush patterns

    • reopening file handles

    • metadata cache controls

  2. Is it strictly required to freeze the file topology (pre-create all groups/datasets) if SWMR is not used?

  3. Is the observed behavior expected, or am I misusing the API in a way that could be corrected?

I am not looking for guarantees of seeing the latest data — only to avoid hard failures and metadata corruption errors on the reader side.

Minimal example

I can provide a minimal reproducer (two small programs: writer + reader) if helpful.

Thank you for your time and for maintaining HDF5.
Any clarification on what is fundamentally unsupported vs. potentially workable would be greatly appreciated.

Best regards,
Younes

3

I have a process that performs a simple subset of what you are asking. It has been working properly with all versions of HDF5 for the past 10 years, without SWMR. A single large HDF5 file is written slowly over several hours. The major difference is that your objective is to “read available data”, whereas my much simpler objective is to read a single data set to monitor progress. Here are some details, in case it might be helpful for your scenario.

  • Both reader and writer are relatively simple C-based applications.
  • The writer keeps the file open for the entire duration (as far as I can tell).
  • The dataset to monitor is created at the start of the writing process. Its schema is unchanged for the entire duration.
  • The schema for the rest of the file is also created once at the start, and never changed or expanded. The only file expansion is adding records to all datasets, along the single “time” dimension.
  • The dataset to monitor is the “time coordinate” for the included data records. It is 1-D chunked unlimited, and only appended one “time step” at a time.
  • The dataset to monitor is in the root group.
  • The writer’s scripting application probably flushes after each write, but I am not sure about that.
  • The reader freshly opens the file on each sampling, and closes the file afterward.
  • I found it necessary to disable HDF5 file locking for the reader, on Linux systems.
  • Otherwise the monitoring application is an ordinary C-based reader that does nothing special.