This is my first post in this list. I work at a company specialized in
engineering applications, and we have been using HDF for several years now
and so far we have been really happy with it.
Lately thought we have been trying to track down a memory leak while reading
some large datasets. First we found the memory leak problem while trying to
execute some simulations in our application. Tracking down the memory usage,
we narrowed it down to our in-house routines that read/write HDF files. To
verify if it was a problem with our code (more likely) or a problem in the
HDF library (highly unlikely), we created some sample code that only uses
the HDF library routines, and that reads a file similar to the one where we
originally found the problem. Unfortunately, the memory leak still occurs.
I'm writing here because we are out of ideas on how to try to figure this
problem out, so perhaps you guys can shed some light in the matter and point
us in the right direction.
The file layout we use in this case is (roughly) as follows:
Above, everything is a group, except for the member "values", which is a
50,000 x 1 dataset of doubles.
As you can see, we have 501 "Timestep" root groups, each containing 17
datasets. We try to read this as follows:
1. Pre-allocate 17 buffers, each one being able to accommodate an entire
2. Go over each time-step, and read the 17 buffers.
Measuring the memory during each Timestep read (i.e., the reading of the 17
datasets inside that Timestep), the memory keeps accumulating, until by the
end of the read of the last Timestep it is over 100Mb. Since we use always
the same buffers, we have no idea of what the problem is. The sample routine
we are using for reading is as follows:
void Read( std::string p_name, void* buffer )
// error checking suppressed for simplicity
hid_t id = H5Dopen2( m_file_id, p_name.c_str(), H5P_DEFAULT ); //
m_file_id is the id of the file, obtained by H5Fopen
H5Dread( id, H5T_NATIVE_DOUBLE, H5S_ALL, H5S_ALL, H5P_DEFAULT,
H5Dclose( id );
We tried even using the same buffer for all datasets, but we still get the
same amount of memory leak.
Does anyone have any idea of what we may be doing wrong in here?
Thanks and Cheers,
ESSS - Engineering Simulation and Scientific Software