Difficulty Reading Large HDF5 Files on Multi-Node Systems

HDF5 Library C++ Users' Group
1

Hello HDF5 Community,

I’m encountering an issue with reading large HDF5 files (500GB+) on a multi-node HPC system. Specifically, when using parallel I/O with MPI, I’ve noticed:

  1. Slow Read Performance: The read times are significantly slower than expected, even with optimized chunking and collective I/O enabled.
  2. Occasional Errors: Intermittent H5Dread() errors occur when accessing datasets with millions of rows.

Here’s my setup:

  • HDF5 Version: 1.12.2
  • MPI Library: OpenMPI 4.1.5
  • File System: Lustre

I’ve tried tuning H5Pset_dxpl_mpio() parameters and adjusting chunk sizes, but the improvements are minimal. Could this be an issue with Lustre striping or MPI-IO settings?

Any advice or insights would be greatly appreciated!

Thank you,

1 Like
2

I recommend updating to HDF5 version 1.14.

Additionally, poor performance can result from various issues. I suggest gathering I/O performance statistics by starting with Darshan’s reports. A helpful introduction to Darshan, MPI I/O, and HDF5 performance can be found in the talks from ANL’s Extreme Computing Training Agenda 2024 « Argonne Training Program on Extreme-Scale Computing.

For Lustre file systems, start with setting the stripe count to -1 and check if that improves performance. It’s also possible that the chunks are not aligning properly with the Lustre striping.

If you have a reproducer for the second issue, please report it at GitHub · Where software is built so we can track it.