Hello
First I want to give a short explanation how I use libhdf5: I have an application which produces a good amount of data which is then stored in multiple hdf5-files. The amount of files is around 300. And for each file there are 30 datasets. Those are always either 1 or 2-dimensional and only the first dimension is allowed to grow. So overall ~9000 datasets are written. Also inside one file all datasets have the same first dimension (=data is always written to all of them).
All those files are rotated after one hour: Existing data is flushed, the hdf5-file is closed and finally a new hdf5-file is opened. It’s important that data is only written / appended.
Due to the global lock in libhdf5 it is not possible to write the data as fast as it is produced. The solution here is that I simply load the library multiple times with dlopen and then one library instance handles 50 files (=1500 datasets). Spawning multiple processes would also be possible, but it has some significant downsides, because the application shares a lot of internal state which would have to be duplicated (=higher memory and CPU usage). Because of that, the dlopen-“trick” is used.
Until today I’ve used libhdf5 1.10.0, but for testing I’ve used now 1.14.4.1.
Now my problem:
The memory usage seems to grow a lot with the number of datasets. In general it seems to grow slowly over the first hour and then it stays roughly the same. I reduced the already not so big chunk-size plus as well the cache-size (both a lot), but the memory usage still stays high (roughly unchanged). Also after rotation I call H5garbage_collect and then I would expect that the memory usage is roughly again what it was initially. However, it keeps being high. If I reduce for testing the number of datasets, then I can see that the memory consumption also reduced quite a lot.
One test I did was to “merge” most of the datasets: E.g. instead of having 10 vectors of size n, I merged them to one matrix with the dimensions of nx10. Plus as well a chunk and cache size which is 10 times larger was used (so overall identical to the sum of the chunk sizes of the vectors). Finally I only kept two datasets, but they contained all the data of the initial 30 datasets. I would expect a similar memory usage, but with this it was already much lower. Unfortunately, I cannot do this merging for in the final code.
Is there some internal memory in libhdf5 which gets allocated and never freed (but reused)? And maybe this amount is roughly proportional to the maximum number of datasets which were opened at a time? If something like that exists, can it be freed somehow? Or reduced?
Thanks a lot
Benjamin Meier
