A lot of datasets in one hdf5 file -> rapidly increasing memory

Hdf-forum archives
1

Hello everyone,

I’m planning to use hdf5 to store a lot of image datasets with some metadata
in one hdf5 file. Ideally there will be a lot of groups, each group will
contain one frame consisting of different images and some metadata.

I tried different ways to realize this, but in the end there is always one
problem: There seemed to be something like a memory leak. With every
iteration (create, write, close a dataset or group) the main memory
increases rapidly.

I found only one solution, close the library by “H5close()”. But this isn’t
a usable way for me because it seems too much of time.

Some information about what I use:
HDF5 1.8.10 or 1.8.9
Visual Studio 2010
Windows 7

As mentioned before, I tried a lot of things:

· Use High level and low level API
· Use garbage collector with property lists
· Use “close” commands in every iteration for datasets, spaces and
file
· Use flush or something similar
· Trying to manually free the memory
· Some other stuff

My questions are:
How can I avoid the library to allocate so much working memory?
How can I free the memory in every iteration?
Would you store these data in another way? Maybe there is a better way to
store a lot of groups with many images and metadata in hfd5.

I attached a simple program to show the problem. It only opens one hdf5 and
writes some datasets. As you can see the memory will increase rapidly.

Thanks a lot in advanced
Nils

···

*******************************************************************

Sample code:

       hid_t file_id, dataset, intType, dataspace;
       hsize_t dims[RANK]={2,3};
       int data[6]={1,2,3,4,5,6};
       herr_t status;
       int count = 0;

       //for dataset name
       char datasetName[256];

       intType= H5Tcopy(H5T_NATIVE_INT);
       dataspace = H5Screate_simple(RANK, dims, NULL);

       /* create a HDF5 file */
       file_id = H5Fcreate ("hdf_test.h5", H5F_ACC_TRUNC, H5P_DEFAULT,
H5P_DEFAULT);

       for(int i = 0; i <= 32000; i++){
             sprintf(datasetName, "/dset_%05d", i);
             //dataset = H5Gcreate(file_id, datasetName.str().c_str(),
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);

             //Create integer type dataset
             dataset = H5Dcreate(file_id, datasetName, intType, dataspace,
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);

             //Release hdf5 dataset and flush file
             H5Fflush(file_id,H5F_SCOPE_GLOBAL);
             H5Dclose(dataset);

             Sleep(100);
             count++;

       }

       /* close file */
       status = H5Fclose (file_id);

--
View this message in context: http://hdf-forum.184993.n3.nabble.com/A-lot-of-datasets-in-one-hdf5-file-rapidly-increasing-memory-tp4025652.html
Sent from the hdf-forum mailing list archive at Nabble.com.

2

7 posts were merged into an existing topic: A lot of datasets in one hdf5 file -> rapidly increasing memory

3