Hello everyone,
I’m planning to use hdf5 to store a lot of image datasets with some metadata
in one hdf5 file. Ideally there will be a lot of groups, each group will
contain one frame consisting of different images and some metadata.
I tried different ways to realize this, but in the end there is always one
problem: There seemed to be something like a memory leak. With every
iteration (create, write, close a dataset or group) the main memory
increases rapidly.
I found only one solution, close the library by “H5close()”. But this isn’t
a usable way for me because it seems too much of time.
Some information about what I use:
HDF5 1.8.10 or 1.8.9
Visual Studio 2010
Windows 7
As mentioned before, I tried a lot of things:
· Use High level and low level API
· Use garbage collector with property lists
· Use “close” commands in every iteration for datasets, spaces and
file
· Use flush or something similar
· Trying to manually free the memory
· Some other stuff
My questions are:
How can I avoid the library to allocate so much working memory?
How can I free the memory in every iteration?
Would you store these data in another way? Maybe there is a better way to
store a lot of groups with many images and metadata in hfd5.
I attached a simple program to show the problem. It only opens one hdf5 and
writes some datasets. As you can see the memory will increase rapidly.
Thanks a lot in advanced
Nils
···
*******************************************************************
Sample code:
hid_t file_id, dataset, intType, dataspace;
hsize_t dims[RANK]={2,3};
int data[6]={1,2,3,4,5,6};
herr_t status;
int count = 0;
//for dataset name
char datasetName[256];
intType= H5Tcopy(H5T_NATIVE_INT);
dataspace = H5Screate_simple(RANK, dims, NULL);
/* create a HDF5 file */
file_id = H5Fcreate ("hdf_test.h5", H5F_ACC_TRUNC, H5P_DEFAULT,
H5P_DEFAULT);
for(int i = 0; i <= 32000; i++){
sprintf(datasetName, "/dset_%05d", i);
//dataset = H5Gcreate(file_id, datasetName.str().c_str(),
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
//Create integer type dataset
dataset = H5Dcreate(file_id, datasetName, intType, dataspace,
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
//Release hdf5 dataset and flush file
H5Fflush(file_id,H5F_SCOPE_GLOBAL);
H5Dclose(dataset);
Sleep(100);
count++;
}
/* close file */
status = H5Fclose (file_id);
--
View this message in context: http://hdf-forum.184993.n3.nabble.com/A-lot-of-datasets-in-one-hdf5-file-rapidly-increasing-memory-tp4025652.html
Sent from the hdf-forum mailing list archive at Nabble.com.