Hello everyone,
I’m planning to use hdf5 to store a lot of image datasets with some metadata in one hdf5 file. Ideally there will be a lot of groups, each group will contain one frame consisting of different images and some metadata.
I tried different ways to realize this, but in the end there is always one problem: There seemed to be something like a memory leak. With every iteration (create, write, close a dataset or group) the main memory increases rapidly.
I found only one solution, close the library by “H5close()”. But this isn’t a usable way for me because it seems too much of time.
Some information about what I use:
HDF5 1.8.10 or 1.8.9
Visual Studio 2010
Windows 7
As mentioned before, I tried a lot of things:
• Use High level and low level API
• Use garbage collector with property lists
• Use “close” commands in every iteration for datasets, spaces and file
• Use flush or something similar
• Trying to manually free the memory
• Some other stuff
My questions are:
How can I avoid the library to allocate so much working memory?
How can I free the memory in every iteration?
Would you store these data in another way? Maybe there is a better way / strategy to store a lot of groups with many images and metadata in hfd5.
I attached a simple program to show the problem. It only opens one hdf5 and writes some datasets (to simplify with small datasets and without groups). As you can see the memory will increase rapidly.
Thanks a lot in advanced!
Nils
···
*********************************************
Sample code:
hid_t file_id, dataset, intType, dataspace;
hsize_t dims[RANK]={2,3};
int data[6]={1,2,3,4,5,6};
herr_t status;
int count = 0;
//for dataset name
char datasetName[256];
intType= H5Tcopy(H5T_NATIVE_INT);
dataspace = H5Screate_simple(RANK, dims, NULL);
/* create a HDF5 file */
file_id = H5Fcreate ("hdf_test.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
for(int i = 0; i <= 32000; i++){
sprintf(datasetName, "/dset_%05d", i);
//dataset = H5Gcreate(file_id, datasetName.str().c_str(), H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
//Create integer type dataset
dataset = H5Dcreate(file_id, datasetName, intType, dataspace, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
//Release hdf5 dataset and flush file
H5Fflush(file_id,H5F_SCOPE_GLOBAL);
H5Dclose(dataset);
Sleep(100);
count++;
}
/* close file */
status = H5Fclose (file_id);