Compiling & using HDF5 from HDF-Forum


#1

Hello together,
I want to plan to using HDF5 from this forum directly on our cluster.
My question now is:
Did I compile the libraries, etc too?

The reason is, that SLURM can using HDF5 and they search for the dev-packet from them.
I am very happy, if everyone can help me here :slight_smile:

Best regards

Zhéue M.


#2

SLURM may be built with HDF5, then used for profiling as outlined here. This requires SLURM compiled with HDF5, and is the optional but not the general usage pattern for the HDF5 libraries on a cluster.

Depending on your filesystem setup you have options:

  • parallel FS: HDF5 should be compiled with parallel support: ./configure --enable-parallel and here is a A Brief Introduction to Parallel HDF5 by Quincey Koziol & Scot Breitenfeld (171 slides). This approach gives option to collective IO, where all processes can write into the same dataset concurrently.
  • POSIX serial FS: this is the default setting with configure, each process must have separate file to write at a time.

The advantage of parallel FS over regular POSIX FS such as EXT4 are: aggregated bandwidth, collective IO where you can write into single dataset resulting in a cleaner code.