Concurrent read while writing HDF5 file (without SWMR): intermittent metadata / object header errors — any supported workaround?

HDF5 Library
1

Hello

I am trying to read an HDF5 file while it is being written by another process, without using SWMR mode. I understand that this may not be officially supported, but I would like to understand whether there is any safe or semi-safe workaround for my use case, or whether this is fundamentally impossible.

I will describe the setup, what works initially, and the errors I eventually hit.

System / environment

  • Platform: Windows 10 x64

  • Compiler: MSVC (Visual Studio 2022)

  • HDF5 version: 1.14.5 (from Conan)

  • Language: C++ (HDF5 C++ API + some direct C API calls)

  • Storage: local NTFS filesystem

  • Processes: separate writer process and reader process (no MPI)

Use case

  • One process writes time steps incrementally into an HDF5 file.

  • New groups, subgroups, and datasets are created as the simulation progresses.

  • Another process repeatedly opens the file, reads available data, then closes it.

  • The reader does not keep the file handle open permanently.

  • No SWMR mode (this is intentional for now).

Expected behavior (best case):

  • Reader may fail to see the newest data, but should not crash.

  • On retry, reader should eventually see newly added groups/datasets.

What I tried

Reader side

  • File opened read-only (H5F_ACC_RDONLY)

  • Tried:

    • disabling file locking

    • opening a fresh file handle on every retry

    • exponential backoff between retries

  • No cached HDF5 objects kept between iterations

Writer side

  • File opened with H5F_ACC_RDWR or created with H5F_ACC_TRUNC

  • Groups and datasets are created dynamically

  • H5Fflush(..., H5F_SCOPE_GLOBAL) is called after writes

Initially, this sometimes works:

  • Reader can read the file correctly for a few iterations

  • New time steps appear as expected

Errors encountered

After some time, the reader fails with hard HDF5 errors, not just “object not found”.

Example 1 — dataset read failure

H5Dread(): can't synchronously read data
...
H5FD_read(): addr overflow, addr = 3299221668, size = 8, eoa = 2922366596
major: Invalid arguments to routine
minor: Address overflowed

Example 2 — object header / metadata corruption (most worrying)

H5O_get_info(): unable to load object header
H5C__verify_len_eoa(): address of object past end of allocation
major: Object cache
minor: Bad value

At this point the reader crashes consistently even after the writer finishes, and I stay on the same process of the loop to try to read each time.

My understanding so far

From reading documentation and debugging:

  • These errors appear when the reader sees partially written metadata

  • The file’s EOA (end of allocation) changes while the reader is accessing metadata

  • HDF5 correctly rejects inconsistent object headers

My question

I fully understand that SWMR is the supported solution, but before fully restructuring my pipeline, I would like to ask:

  1. Is there any documented or undocumented workaround to make this pattern safer without SWMR?

    • e.g. specific flush patterns

    • reopening file handles

    • metadata cache controls

  2. Is it strictly required to freeze the file topology (pre-create all groups/datasets) if SWMR is not used?

  3. Is the observed behavior expected, or am I misusing the API in a way that could be corrected?

I am not looking for guarantees of seeing the latest data — only to avoid hard failures and metadata corruption errors on the reader side.

Minimal example

I can provide a minimal reproducer (two small programs: writer + reader) if helpful.

Thank you for your time and for maintaining HDF5.
Any clarification on what is fundamentally unsupported vs. potentially workable would be greatly appreciated.

Best regards,
Younes

3

I have a process that performs a simple subset of what you are asking. It has been working properly with all versions of HDF5 for the past 10 years, without SWMR. A single large HDF5 file is written slowly over several hours. The major difference is that your objective is to “read available data”, whereas my much simpler objective is to read a single data set to monitor progress. Here are some details, in case it might be helpful for your scenario.

  • Both reader and writer are relatively simple C-based applications.
  • The writer keeps the file open for the entire duration (as far as I can tell).
  • The dataset to monitor is created at the start of the writing process. Its schema is unchanged for the entire duration.
  • The schema for the rest of the file is also created once at the start, and never changed or expanded. The only file expansion is adding records to all datasets, along the single “time” dimension.
  • The dataset to monitor is the “time coordinate” for the included data records. It is 1-D chunked unlimited, and only appended one “time step” at a time.
  • The dataset to monitor is in the root group.
  • The writer’s scripting application probably flushes after each write, but I am not sure about that.
  • The reader freshly opens the file on each sampling, and closes the file afterward.
  • I found it necessary to disable HDF5 file locking for the reader, on Linux systems.
  • Otherwise the monitoring application is an ordinary C-based reader that does nothing special.
1 Like
4

I’m getting back to this a bit late, but the description you originally gave is very similar to one that we’ve been investigating ourselves: “One process writes the time steps incrementally into an HDF5 file, creating new groups, subgroups, and datasets as the simulation proceeds, while another process opens the file repeatedly, reads the available data, and then closes it.”

The workaround mentioned above (i.e., pre-create the complete schema, add only along the time dimension, and disable locking) does work. However, it exactly gives up the feature you wanted: dynamic structure as the simulation proceeds. We are currently developing a PoC HDF5 VOL connector (called vol-stream) that includes step/transaction semantics for this kind of situation, and we have a small PoC designed to address part of your problem: allowing new groups to appear safely to a second process without giving up dynamic structure.

When VFD SWMR is used, the test writer carries out 12 consecutive steps, adding to a growing dataset (with new groups named /step// appearing on the disk as this takes place), while a separate reader, using plain native HDF5 and not our connector, opens the file once and continuously checks it during the time the writer is still writing. In the absence of VFD SWMR, the reader simply does not see the new structure at all (without any error, it is just invisible); but when VFD SWMR is employed, in a single test run the reader is able to observe the structure growing live over all 12 steps, with each piece of content being correctly read.

I need to make it clear what this does and what it doesn’t: our reader leaves the file open between polls rather than closing and reopening it each time, which is a bit different from the situation you described. Moreover, the failure that you’ve been able to reproduce in the absence of VFD SWMR is silent invisibility, not the object-header corruption/address-overflow errors that you actually encountered. I can only say that it achieves the same fundamental objective (ensuring that the structure and data can be viewed safely while the writer still has the file open).

Most importantly, again, the VOL is still just a proof-of-concept. It does not handle parallel writers, and long-running stability is not guaranteed. Moreover, it at present relies on VFD SWMR, which hasn’t yet been included in the mainline HDF5 2.x.

Does the goal of building the structure while keeping it open here match closely enough that it’s worth trying against your real workload as this gets further developed? And is the pattern of opening once and then polling suitable for you, or do you specifically require the close/reopen approach?

1 Like