# Difficulty Reading Large HDF5 Files on Multi-Node Systems

**URL:** <https://forum.hdfgroup.org/t/difficulty-reading-large-hdf5-files-on-multi-node-systems/12824>\
**Category:** C++ Users' Group\
**Created:** [November 27, 2024, 3:41am UTC](https://forum.hdfgroup.org/t/difficulty-reading-large-hdf5-files-on-multi-node-systems/12824 "2024-11-27T03:41:14Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sheetgb](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/s/858c86/32.png) [@sheetgb](https://forum.hdfgroup.org/u/sheetgb)\
**Post date:** [November 27, 2024, 3:41am UTC](https://forum.hdfgroup.org/t/difficulty-reading-large-hdf5-files-on-multi-node-systems/12824/1 "2024-11-27T03:41:14Z")

</div>

Hello HDF5 Community,

I’m encountering an issue with reading large HDF5 files (500GB+) on a multi-node HPC system. Specifically, when using parallel I/O with MPI, I’ve noticed:

1. **Slow Read Performance:** The read times are significantly slower than expected, even with optimized chunking and collective I/O enabled.
2. **Occasional Errors:** Intermittent `H5Dread()` errors occur when accessing datasets with millions of rows.

Here’s my setup:

- HDF5 Version: 1.12.2
- MPI Library: OpenMPI 4.1.5
- File System[:](https://menuland.ph/shakeys-menu-prices-in-philippines/) Lustre

I’ve tried tuning `H5Pset_dxpl_mpio()` parameters and adjusting chunk sizes, but the improvements are minimal. Could this be an issue with Lustre striping or MPI-IO settings?

Any advice or insights would be greatly appreciated!

Thank you,

---

<div class="post-metadata">

**Author:** ![brtnfld](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/brtnfld/32/1444_2.png) [@brtnfld](https://forum.hdfgroup.org/u/brtnfld)\
**Post date:** [December 3, 2024, 6:17pm UTC](https://forum.hdfgroup.org/t/difficulty-reading-large-hdf5-files-on-multi-node-systems/12824/2 "2024-12-03T18:17:29Z")

</div>

I recommend updating to HDF5 version 1.14.

Additionally, poor performance can result from various issues. I suggest gathering I/O performance statistics by starting with Darshan’s reports. A helpful introduction to Darshan, MPI I/O, and HDF5 performance can be found in the talks from ANL’s Extreme Computing Training [Agenda 2024 « Argonne Training Program on Extreme-Scale Computing](https://extremecomputingtraining.anl.gov/agenda-2024/#Track-7).

For Lustre file systems, start with setting the stripe count to -1 and check if that improves performance. It’s also possible that the chunks are not aligning properly with the Lustre striping.

If you have a reproducer for the second issue, please report it at [GitHub · Where software is built](https://github.com/HDFGroup/hdf5/issues) so we can track it.