Hi everyone,
I’m facing a kind of memory leak when writing vlen data into a chunked dataset.
I create a compound datatype that contains 2 vlen types. Then I continuously create a file, add a chunked dataset of compound type (including vlen arrays), write 10 elements in the dataset, and close everything.
While doing so, I observed the memory usage of my program that keeps growing. Calls to H5close() can solve this, but I would prefer to avoid it. It might also be a bad usage of the library from my side.
Below is a sample code to reproduce this:
typedef struct TimeData
{
int64_t timestamp;
int64_t flags;
hvl_t data;
hvl_t attrs;
} TimeData;
int main(int argc, char** argv)
{
// Infinit loop that continuously create a file,
// add a chunked dataset of compound type (including vlen arrays),
// write 10 elements in the dataset, and close everything.
//
while (true)
{
// Create compound data type
hid_t id = 0;
id = H5Tcreate(H5T_COMPOUND, sizeof(TimeData));
H5Tinsert(id, "time", HOFFSET(TimeData, timestamp), H5T_NATIVE_INT64);
H5Tinsert(id, "flags", HOFFSET(TimeData, flags), H5T_NATIVE_INT64);
H5Tinsert(id, "data", HOFFSET(TimeData, data), H5Tvlen_create(H5T_NATIVE_UINT8));
H5Tinsert(id, "attrs", HOFFSET(TimeData, attrs), H5Tvlen_create(H5T_NATIVE_UINT8));
// Create file
hid_t file = H5Fcreate("data2.h5", H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
hid_t set = H5I_INVALID_HID;
// Create chunked data set
hsize_t dims[] = { 0 };
hsize_t max_dims[] = { H5S_UNLIMITED };
hid_t space = H5Screate_simple(1, dims, max_dims);
hid_t prop = H5Pcreate(H5P_DATASET_CREATE);
hsize_t chunk_dim[] = { 32 };
H5Pset_chunk(prop, 1, chunk_dim);
set = H5Dcreate(file, "/video", id, space, H5P_DEFAULT, prop, H5P_DEFAULT);
H5Tclose(id);
H5Sclose(space);
H5Pclose(prop);
// Extend and write 10 entries
for (int i = 0; i < 10; ++i)
{
// extend dataset
hsize_t size[] = { i + 1 };
H5Dset_extent(set, size);
// select hyperslab and write as VL array
hid_t set_space = H5Dget_space(set);
hsize_t offset[] = { i };
hsize_t dim[] = { 1 };
H5Sselect_hyperslab(set_space, H5S_SELECT_SET, offset, NULL, dim, NULL);
hid_t packet_space = H5Screate_simple(1, dim, dim);
hid_t dataset_type = H5Dget_type(set);
// Write the data with empty ven arrays
TimeData d;
memset(&d, 0, sizeof(d));
H5Dwrite(set, dataset_type, packet_space, set_space, H5P_DEFAULT, &d);
H5Sclose(set_space);
H5Sclose(packet_space);
H5Tclose(dataset_type);
}
H5Dclose(set);
H5Fclose(file);
// Memory leak without calls to this?
// H5close();
}
return 0;
}
The leak seems to happen when closing the file/dataset. Indead, writing a lot of data (instead of 10) in the file does not change the leaked amount.
Bests
Victor