How to correctly use Application Mapping Options

HDF5 Library
1

Context

We have an application that makes use of the hdf5 library. So far, we have build this application using 1.8.21 (on Linux, x86_64, compiled with conan and the default options for a static library). We recently migrated from gcc-11 to gcc-14 and when compiling 1.8.21 with gcc-14 you are met with lots of compilation errors… So, we decided it might be worthwhile to upgrade the hdf5 library.

We picked the current release, i.e. 1.14.5 and building it was no issue. Our use case is not incredibly complicated, but we have a custom plugin as well as a C++ convenience wrapper around some of the hdf5 functions. Everything worked as we intended.

First try

During this migration the question about backwards/forwards compatibility came up and I came across this page. The goal is simple in theory: users who use our application compiled with hdf5-1.8.21 (SW_OLD) and the ones that use hdf5-1.14.5 (SW_NEW) should be able to read and write files (FILE_OLD created by SW_OLD and FILE_NEW created with SW_NEW), that can be read by both versions of the software (since it will take some time until all users have updated to a version of our app that uses 1.14.5).

After reading the Application Mapping Options, I thought the approach seems clear:

  1. Compile hdf5 and make sure that HDF5_ENABLE_DEPRECATED_SYMBOLS is set to ON
  2. Compile our app and make sure that we pass -DH5_USE_18_API as compile option when building our application.
  3. This should be it.

Problems

Testing this revealed crashes. When trying to read FILE_NEW with OLD_SW the closing of the files resulted in memory corruption (double free, etc.).

Im currently suspecting that we are not correctly compiling the library/our application (since in principle what we want should be possible, right?). Looking at the CMakeLists.txt file, there seem to be quite a few options and Im not sure which ones are necessary and which are deduced…

I have seen:

  • HDF5_ENABLE_DEPRECATED_SYMBOLS (bool)
  • HDF5_DEFAULT_API_VERSION ([v16, v18, v110, v112, v114, v200])
  • H5_USE_18_API_DEFAULT (bool)
  • H5_USE_18_API (bool)

Questions:

  • Is what we want possible, in principle?
  • From the docs it seems clear that HDF5_ENABLE_DEPRECATED_SYMBOLS and HDF5_DEFAULT_API_VERSION, H5_USE_18_API_DEFAULT need to be set when compiling the library, but what about H5_USE_18_API ?
  • When compiling our application, we only need to set H5_USE_18_API correct?
  • The article mentions h5cc, but we are just compiling with g++-14. Are there any other things that are needed here?
1 Like
2

Yes, this is the right approach, a little discussion of what is happening may help.
First there are two parts - the HDF5 library and the application.

HDF5 Library:

  • HDF5_ENABLE_DEPRECATED_SYMBOLS (bool) (ON by default)
  • HDF5_DEFAULT_API_VERSION ([v16, v18, v110, v112, v114, v200]) (1.14.5 defaults to v114)
    So to build HDF5 as a drop in replacement for the 1.8.21 library you need to set;
    HDF5_DEFAULT_API_VERSION:STRING=v18
    Then you should be able to use the install with your existing application (no recompile or relinking).

To rebuild an application that uses 1.8 APIs with a new HDF5 1.14.5 library (with/without HDF5_DEFAULT_API_VERSION set) build the application with define H5_USE_18_API=ON;
set (H5_USE_18_API ON)
So that the application in a cmake build would have:
target_compile_options(app_target_name PRIVATE -DH5_USE_18_API)

The HDF5 repo github workflow, main-cmake-spc.yml, has examples for building the library.
The HDF5Examples folder/project uses the H5_USE_xx_API defines.

Allen

1 Like
3

It’s important to clarify that the API-related functions/variables mentioned are primarily for maintaining compatibility with the source code APIs. They are designed to help you avoid addressing potential API changes that might have occurred in later library releases. However, it’s essential to note that these functions/variables are unrelated to ensuring backward compatibility with the file format. For that specific task, you should use H5Pset_libver_bounds. It is a way to guarantee that a file generated with a newer HDF5 version is readable by a 1.8 library-built application; for that case, it would be H5Pset_libver_bounds(H5F_LIBVER_V18, H5F_LIBVER_V18).

1 Like
4

Thanks so much, this clarified the build process!

5

Thank you for taking the time to answer and pointing this out!

Just to clarify here: so far we have been creating the H5 file to write our data into like this

std::string name{"filename.h5"};
unsigned int flags{ H5_ACC_TRUNC };

H5::H5File file{ name.c_str(), flags };

and to accomodate this new shift we would need to do the following:

std::string name{"filename.h5"};
unsigned int flags{ H5_ACC_TRUNC };

hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
H5Pset_libver_bounds(fapl, H5F_LIBVER_V18, H5F_LIBVER_V18);

H5::H5File file{ name.c_str(), flags, H5::FileAccPropList{ fapl } };

Similarly, we would need to change the code for the reading of the file and make sure that the File Access Properties have been set correctly to read the file (essentially calling H5Pset_libver_bounds again)?

P.S. If there is a overload of the function that accepts only two arguments I apologize… I just extrapolated from the link you provided and assumed one needs to do this over H5::FileAccPropList…

6

That is the correct procedure for the fapl.
You don’t need to set it to read the file; you only need it when writing to the file.