Bug: Library compatibility version differs between CMake and autotools build on macOS

Building 1.10.4 using autotools, the encoded library compatibility version is 104.0.0, while building the same version with CMake, the encoded library compatibility version is 103.0.0. This means a program linked against an autotools-built HDF5 cannot run with a CMake-built HDF5, even if the HDF5 version is the same.

For example, this is the HDF5 1.10.4 that is included with the h5py 2.9.0 macOS wheel, built using autotools:

(env) dirac:hdf5-1.10.4 insight$ otool -l ~/env/lib/python3.7/site-packages/h5py/.dylibs/libhdf5.103.dylib | grep -A 5 LC_ID_DYLIB
          cmd LC_ID_DYLIB
      cmdsize 56
         name /DLC/h5py/libhdf5.103.dylib (offset 24)
   time stamp 1 Thu Jan  1 01:00:01 1970
      current version 104.0.0
compatibility version 104.0.0

While this is the same version (1.10.4) that I’ve built with CMake:

(env) dirac:hdf5-1.10.4 insight$ otool -l ~/Insight/HDF5-1.10.4-Darwin/HDF_Group/HDF5/1.10.4/lib/libhdf5.103.dylib | grep -A 5 LC_ID_DYLIB
          cmd LC_ID_DYLIB
      cmdsize 56
         name @rpath/libhdf5.103.dylib (offset 24)
   time stamp 1 Thu Jan  1 01:00:01 1970
      current version 103.0.0
compatibility version 103.0.0
(env) dirac:hdf5-1.10.4 insight$

I believe this discrepancy is due to how the HDF5 CMake build calculates the SOVERSION, which is what CMake uses as -compatibility_version to the linker.

From CMakeLists.txt:

#-----------------------------------------------------------------------------
# parse the full soversion number from config/lt_vers.am and include in H5_SOVERS_INFO
#-----------------------------------------------------------------------------
file (READ ${HDF5_SOURCE_DIR}/config/lt_vers.am _lt_vers_am_contents)
string (REGEX REPLACE ".*LT_VERS_INTERFACE[ \t]+=[ \t]+([0-9]*).*$"
    "\\1" H5_LIB_SOVERS_INTERFACE ${_lt_vers_am_contents})
string (REGEX REPLACE ".*LT_VERS_REVISION[ \t]+=[ \t]+([0-9]*).*$"
    "\\1" H5_LIB_SOVERS_MINOR ${_lt_vers_am_contents})
string (REGEX REPLACE ".*LT_VERS_AGE[ \t]+=[ \t]+([0-9]*).*$"
    "\\1" H5_LIB_SOVERS_RELEASE ${_lt_vers_am_contents})
math (EXPR H5_LIB_SOVERS_MAJOR ${H5_LIB_SOVERS_INTERFACE}-${H5_LIB_SOVERS_RELEASE})
message (STATUS "SOVERSION: ${H5_LIB_SOVERS_MAJOR}.${H5_LIB_SOVERS_RELEASE}.${H5_LIB_SOVERS_MINOR}")

so LT_VERS_INTERFACE = 103 and LT_VERS_AGE = 0 from config/lt_vers.am are parsed into H5_LIB_SOVERS_INTERFACE and H5_LIB_SOVERS_RELEASE respectively, and H5_LIB_SOVERS_MAJOR is set to H5_LIB_SOVERS_INTERFACE - H5_LIB_SOVERS_RELEASE = 103.

Later in CMakeLists.txt, this H5_LIB_SOVERS_MAJOR is assigned to HDF5_LIB_PACKAGE_SOVERSION_MAJOR and then we have this snippet in config/cmake/HDF5Macros.cmake:

  if (${libtype} MATCHES "SHARED")
    set (PACKAGE_SOVERSION ${HDF5_${libpackage}_PACKAGE_SOVERSION})
    if (WIN32)
      set (LIBHDF_VERSION ${HDF5_PACKAGE_VERSION_MAJOR})
    else ()
      set (LIBHDF_VERSION ${HDF5_${libpackage}_PACKAGE_SOVERSION_MAJOR})
    endif ()
    set_target_properties (${libtarget} PROPERTIES VERSION ${PACKAGE_SOVERSION})
    if (WIN32)
        set (${LIB_OUT_NAME} "${LIB_OUT_NAME}-${LIBHDF_VERSION}")
    else ()
        set_target_properties (${libtarget} PROPERTIES SOVERSION ${LIBHDF_VERSION})
    endif ()
  endif ()

Here the SOVERSION is set to 103, and from https://cmake.org/cmake/help/v3.12/prop_tgt/SOVERSION.html :

For shared libraries and executables on Mach-O systems (e.g. OS X, iOS), the SOVERSION property corresponds to compatibility version and VERSION to current version.

I’m not sure what the real bug is actually. Should the LT_VERS_INTERFACE in config/lt_vers.am really be 103 for the 1.10.4 release? I haven’t looked at the autotools side of things, so I’m not sure if/how that value is used there.

I had a look at the autotools stuff, and I believe this snippet from bin/ltmain.sh explains it:

darwin)
  # Like Linux, but with the current version available in
  # verstring for coding it into the library header
  func_arith $current - $age
  major=.$func_arith_result
  versuffix=$major.$age.$revision
  # Darwin ld doesn't like 0 for these options...
  func_arith $current + 1
  minor_current=$func_arith_result
  xlcverstring="$wl-compatibility_version $wl$minor_current $wl-current_version $wl$minor_current.$revision"
  verstring="-compatibility_version $minor_current -current_version $minor_current.$revision"
  # On Darwin other compilers
  case $CC in
      nagfor*)
          verstring="$wl-compatibility_version $wl$minor_current $wl-current_version $wl$minor_current.$revision"
          ;;
      *)
          verstring="-compatibility_version $minor_current -current_version $minor_current.$revision"
          ;;
  esac
  ;;

Libtool doing + 1 here probably explains why you’ve chosen to set LT_VERS_INTERFACE to 103 on HDF5 1.10.4?

Then maybe the CMake logic should be updated to match the libtool logic, so add 1?

As a side note, I found this snipped just below in config/cmake/HDF5Macros.cmake:

  #-- Apple Specific install_name for libraries
  if (APPLE)
    option (HDF5_BUILD_WITH_INSTALL_NAME "Build with library install_name set to the installation path" OFF)
    if (HDF5_BUILD_WITH_INSTALL_NAME)
      set_property(TARGET ${libtarget} APPEND PROPERTY
          LINK_FLAGS "-current_version ${HDF5_PACKAGE_VERSION} -compatibility_version ${HDF5_PACKAGE_VERSION}"
      )
      set_target_properties (${libtarget} PROPERTIES
          INSTALL_NAME_DIR "${CMAKE_INSTALL_PREFIX}/lib"
          BUILD_WITH_INSTALL_RPATH ${HDF5_BUILD_WITH_INSTALL_NAME}
      )
    endif ()
    ...

so it seems that currently, one way of working around this discrepancy issue is to make do the CMake build with the HDF5_BUILD_WITH_INSTALL_NAME option set, because then the CMake scripts will manually pass -compatibility_version ${HDF5_PACKAGE_VERSION}. Haven’t tested this though, just an observation.

We have tried to address this for 1.10.5, and from our testing we think we have. However, please let us know the results of your tests.

Allen

Thanks @byrn, will test.

Hi again @byrn and sorry for the (very!) late response.

I just built HDF5 1.10.5 on macOS Mojave using autotools with:

brew install autoconf automake libtool
curl -O https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.5/src/hdf5-1.10.5.tar.bz2
tar xvfj hdf5-1.10.5.tar.bz2
cd hdf5-1.10.5
autoreconf -fiv
./configure \
      --disable-dependency-tracking \
      --disable-silent-rules \
      --prefix=$HOME/hdf5-install \
      --enable-build-mode=production \
      --disable-fortran \
      --disable-cxx
make -j4 install

However, it seems the library compatibility version of the resulting HDF5 library is still set to an unnecessarily high value (105.0.0):

dirac:hdf5-1.10.5 insight$ otool -l $HOME/hdf5-install/lib/libhdf5.103.dylib | grep -A 5 LC_ID_DYLIB
          cmd LC_ID_DYLIB
      cmdsize 80
         name /Users/insight/hdf5-install/lib/libhdf5.103.dylib (offset 24)
   time stamp 1 Thu Jan  1 01:00:01 1970
      current version 105.0.0
compatibility version 105.0.0
dirac:hdf5-1.10.5 insight$

Compare this to a CMake build of HDF5 1.10.6 which I have around, where the library compatibility version is 103.0.0:

dirac:hdf5-1.10.5 insight$ otool -l ~/Insight/HDF5-1.10.6-Darwin/HDF_Group/HDF5/1.10.6/lib/libhdf5.103.dylib | grep -A 5 LC_ID_DYLIB
          cmd LC_ID_DYLIB
      cmdsize 56
         name @rpath/libhdf5.103.dylib (offset 24)
   time stamp 1 Thu Jan  1 01:00:01 1970
      current version 103.2.0
compatibility version 103.0.0
dirac:hdf5-1.10.5 insight$

So there’s still a discrepancy.

The unnecessarily high library compatibility version of the autotools-built HDF5 makes it impossible to use applications/libraries linked against such a HDF5 library with a CMake-built HDF5 library, even if the libraries are in fact compatible, due to the compatibility version checking done by the dynamic loader.