On macOS when using hdf5 through python, I’m getting some crashes with compression filters and HDF5 1.12.0 either lzf embedded in h5py or the one from hdf5plugin.
From the investigation, it seems to lead to a call to free in the compression filters.
This was working with HDF5 1.10.6.
Has anything changed regarding this in 1.12.0?
Where can I find an up-to-date compression filter example?
Ø Apparently one should use H5allocate_memory/H5free_memory rather than malloc/free… which seems to fix the issue… but it doesn’t seem any compression filter is following this.
I recently wrote the JPEG filter plugin that is now part of HDF5. When doing so I explicitly asked the HDF5 Group if I should be using H5allocate_memory/H5free_memory or malloc/free. They told me to use malloc/free.
This is the message from the HDF5 Helpdesk I received on 5/27/2019:
Thanks for the reminder!
I brought this up with the developers and they said you definitely should be using malloc() and free() when writing a plugin filter. You should not use malloc and H5free_memory together.
Using malloc/free means that the application is managing the allocating of memory and the freeing of it.
The H5allocate_memory/H5free_memory APIs should be used together, and when used, mean that the HDF5 library is managing the allocating of memory and freeing of it. There were reasons we determined that the filters should manage the memory, rather than using these APIs.
The developer said he had to make changes to many of the filters when he added them to the hdf5_plugin code. I attached the hdf5_plugin source code to this message. Hopefully, looking at the filters included in the hdf5_plugin source will help to clear up some of the questions (but let us know if you still have questions!).
We really need to document this somewhere, as it is not clear.
Are you building with debug enabled using autotools?
With releases (other than HDF5-1.10.7), if you run in debug mode, the --enable-memory-alloc-sanity-check option is also enabled. This option causes the problem. In HDF5-1.10.7 (and future releases), the default is (will be) to NOT enable the memory allocation sanity check feature when debug mode is specified (unless using development code, such as develop or hdf5_1_10).
If using HDF5-1.10.6 in debug mode, then turn the --enable-memory-alloc-sanity-check option off. Here is the output from ./configure --help regarding this option:
Enable this option to turn on internal memory
allocation sanity checking. This could cause more
memory use and somewhat slower allocation. This
option is orthogonal to the
--enable-using-memchecker option. [default=yes if
debug build in a development branch, otherwise no]
Here is the entry from th HDF5-1.10.7 release notes about the issue:
Disable memory sanity checks in the Autotools in release branches
The library can be configured to use internal memory sanity checking,
which replaces C API calls like malloc(3) and free(3) with our own calls
which add things like heap canaries. These canaries can cause problems
when external filter plugins reallocate canary-marked buffers.
For this reason, the default will be to not use the memory allocation
sanity check feature in release branches (e.g., hdf5_1_10_7).
Debug builds in development branches (e.g., develop, hdf5_1_10) will
still use them by default.
This change only affects Autotools debug builds. Non-debug autotools
builds and all CMake builds do not enable this feature by default.
Could this be the issue?
Sorry for any confusion this has caused!
Should we prefer building with CMake on all platforms? We’re currently building HDF5 with CMake on Windows and autotools on Linux & Mac, but it wouldn’t be too hard to change that if CMake is better supported.