Issue with compression filters and HDF5 1.12.0 on macos

Hi,

On macOS when using hdf5 through python, I’m getting some crashes with compression filters and HDF5 1.12.0 either lzf embedded in h5py or the one from hdf5plugin.
From the investigation, it seems to lead to a call to free in the compression filters.
This was working with HDF5 1.10.6.
Has anything changed regarding this in 1.12.0?
Where can I find an up-to-date compression filter example?

For more details (traceback,…), see:

Apparently one should use H5allocate_memory/H5free_memory rather than malloc/free… which seems to fix the issue… but it doesn’t seem any compression filter is following this.
Even the example bzip2 filter provided by the HDFGroup is not following it (https://confluence.hdfgroup.org/display/support/Registered+Filter+Plugins#RegisteredFilterPlugins-regcomp).
There is a Pull Request for hdf5-blosc but it is still opened: https://github.com/Blosc/hdf5-blosc/pull/21).

This is related to this discussion: Filter plugins and the library: to link or not to link?

Ø Apparently one should use H5allocate_memory/H5free_memory rather than malloc/free… which seems to fix the issue… but it doesn’t seem any compression filter is following this.

I recently wrote the JPEG filter plugin that is now part of HDF5. When doing so I explicitly asked the HDF5 Group if I should be using H5allocate_memory/H5free_memory or malloc/free. They told me to use malloc/free.

This is the message from the HDF5 Helpdesk I received on 5/27/2019:

Hi Mark,

Thanks for the reminder!

I brought this up with the developers and they said you definitely should be using malloc() and free() when writing a plugin filter. You should not use malloc and H5free_memory together.

Using malloc/free means that the application is managing the allocating of memory and the freeing of it.

The H5allocate_memory/H5free_memory APIs should be used together, and when used, mean that the HDF5 library is managing the allocating of memory and freeing of it. There were reasons we determined that the filters should manage the memory, rather than using these APIs.

The developer said he had to make changes to many of the filters when he added them to the hdf5_plugin code. I attached the hdf5_plugin source code to this message. Hopefully, looking at the filters included in the hdf5_plugin source will help to clear up some of the questions (but let us know if you still have questions!).

We really need to document this somewhere, as it is not clear.

-Barbara

Thanks for your answer!

Do you know where I could get the hdf5_plugin code? and if the patches were reported upstream?

I think the issue here is that the filter code (lzf_filter.c) is freeing a buffer (buf) which it doesn’t allocate. Presumably it’s allocated by HDF5 before calling the filter, using H5allocate_memory.

Maybe the filter code shouldn’t be freeing that memory at all, and leaving it for HDF5 to do?

Calling H5free_memory instead of free makes it work, commenting free delays the error to the closing of Python where an assert is raised:

Assertion failed: (0 == H5MM_curr_alloc_bytes_s), function H5MM_final_sanity_check, file H5MM.c, line 232.

Trackeback:

|0   libsystem_kernel.dylib        |0x00007fffdfaa8d42 __pthread_kill + 10|
|---|---|
|1   libsystem_pthread.dylib       |0x00007fffdfb96457 pthread_kill + 90|
|2   libsystem_c.dylib             |0x00007fffdfa0e420 abort + 129|
|3   libsystem_c.dylib             |0x00007fffdf9d5893 __assert_rtn + 320|
|4   libhdf5.200.dylib             |0x000000010de694a7 H5MM_final_sanity_check + 71|
|5   libhdf5.200.dylib             |0x000000010dad8e8b H5_term_library + 10475|
|6   libsystem_c.dylib             |0x00007fffdfa0f17f __cxa_finalize_ranges + 339|
|7   libsystem_c.dylib             |0x00007fffdfa0f4b2 exit + 55|
|8   org.python.python             |0x000000010c7a023c Py_Exit + 28|
|9   org.python.python             |0x000000010c7a8e98 handle_system_exit + 360|
|10  org.python.python             |0x000000010c7a8ac8 PyErr_PrintEx + 56|
|11  org.python.python             |0x000000010c7c7ff0 pymain_run_module + 160|
|12  org.python.python             |0x000000010c7c73e7 pymain_main + 5351|
|13  org.python.python             |0x000000010c7c7f2a _Py_UnixMain + 58|
|14  libdyld.dylib                 |0x00007fffdf97a235 start + 1|

Hello!

Are you building with debug enabled using autotools?

With releases (other than HDF5-1.10.7), if you run in debug mode, the --enable-memory-alloc-sanity-check option is also enabled. This option causes the problem. In HDF5-1.10.7 (and future releases), the default is (will be) to NOT enable the memory allocation sanity check feature when debug mode is specified (unless using development code, such as develop or hdf5_1_10).

If using HDF5-1.10.6 in debug mode, then turn the --enable-memory-alloc-sanity-check option off. Here is the output from ./configure --help regarding this option:

   --enable-memory-alloc-sanity-check
                  Enable this option to turn on internal memory
                   allocation sanity checking. This could cause more
                   memory use and somewhat slower allocation. This
                   option is orthogonal to the
                   --enable-using-memchecker option. [default=yes if
                   debug build in a development branch, otherwise no]

Here is the entry from th HDF5-1.10.7 release notes about the issue:

  • Disable memory sanity checks in the Autotools in release branches

    The library can be configured to use internal memory sanity checking,
    which replaces C API calls like malloc(3) and free(3) with our own calls
    which add things like heap canaries. These canaries can cause problems
    when external filter plugins reallocate canary-marked buffers.
    
    For this reason, the default will be to not use the memory allocation
    sanity check feature in release branches (e.g., hdf5_1_10_7).
    Debug builds in development branches (e.g., develop, hdf5_1_10) will
    still use them by default.
    
    This change only affects Autotools debug builds. Non-debug autotools
    builds and all CMake builds do not enable this feature by default.
    

Could this be the issue?
Sorry for any confusion this has caused!

-Barbara

Hi,

This maybe the issue, but the issue is with HDF5 v1.12.0.
From what I tested with HDF5 1.10.6 it is working fine.

h5py is using the following way for building libhdf5 on macos: https://github.com/h5py/h5py/blob/master/ci/travis/get_hdf5_if_needed.sh#L16-L26
And I am building it the same way.

What are the default options on macos for HDF5 1.12.0?

Hi Thomas,

The issue was not resolved in HDF5-1.12.0 (which came before HDF5-1.10.7). It should be fixed in HDF5-1.12.1 and HDF5-1.8.22.

Could debug be specified somewhere in your build code? (I didn’t see it.)

-Barbara

Hi,

Using --enable-memory-alloc-sanity-check=no fixes the problem.

But it turns out that default build mode of libhdf5 1.12.0 is debug… which is the root of this issue.

Should we prefer building with CMake on all platforms? We’re currently building HDF5 with CMake on Windows and autotools on Linux & Mac, but it wouldn’t be too hard to change that if CMake is better supported.

Yes, I can see that!! I’ll pass this on to the developers.
Thank you!
-Barbara

OK, thank you!

  • Thomas

Hi Thomas1,

CMake is well supported on Linux and Mac, as well as on Windows. We are maintaining support for both autotools and CMake and attempting to make them match as much as possible.

However, there are differences between the two. For example:

  • The h5cc scripts are similar but different.
  • Only autotools can fully test SWMR (we are working on this)
  • Packaging is different

-Barbara

Thanks Barbara! If they’re equally supported, we’ll probably stick with what we’re doing.