Large memory consumption when loading many data sets

Thanks for the quick reply @jhenderson

Gives

---------------------------------

h5py    3.10.0
HDF5    1.14.2
Python  3.9.18 (main, Jan  4 2024, 00:00:00)
[GCC 11.4.1 20230605 (Red Hat 11.4.1-2)]
sys.platform    linux
sys.maxsize     9223372036854775807
numpy   1.24.3
cython (built with) 0.29.36
numpy (built against) 1.19.3
HDF5 (built against) 1.14.2

When I run your test script I get the following output, which seems to be better then the result in the jupyter notebook, but still shows the problem.

$ python test.py
4.470348358154297 GiB
4.47119140625 GiB
8.942459106445312 GiB
5.8122406005859375 GiB