I am developing a multi-threaded C++ application using the C version of the HDF5 library version 1.10.2. I am developing on Red Hat Enterprise Linux 7.7. The application is long-running but uses a lot of memory (~2GB for writing ~100MB HDF5 files). Valgrind does not show any memory leaks.
The main thread spawns up to 16 threads to gather the data from other processes. Then 15 of the threads complete and the remaining thread spawns another thread to write the data to an HDF5 file. The data is 1-D by the way. My application uses up more and more system memory, up to ~2GB after about 2 hours. For another run with more points, the system memory fills up and then the application reads and writes to disk.
The previous version of this application, which was also multi-threaded, did not use HDF5 and did not use as much memory. I tried massif with the option set to show page allocations and the largest page allocation was not for HDF5 but another library used in the application. That other library is used in the previous version of my application but it did use as much memory.
I experimented with various chunk sizes. The original chunk size was 1x10 items (usually 40 B), then I tried increasing the chunk_size to 1x1000 (4kB) and the memory usage stayed the same.