How to build (properly) static (fortran and parllel enabled) HDF5 library

HDF5 Library
1

Dear all,

I am not able to build a “working” version of the HDF5 library in static form (in particular with fortran and parallel enabled, but I think this is not my issue). On the contrary, I can build and use the shared form successfully.

I am using CMake with the following flags:

CC=mpicc CXX=mpicxx FC=mpif90 cmake ../ \
     -DCMAKE_INSTALL_PREFIX=$HDF5_PATH \
     -DBUILD_SHARED_LIBS:BOOL=OFF \
     -DBUILD_STATIC_LIBS:BOOL=ON \
     -DHDF5_BUILD_FORTRAN:BOOL=ON \
     -DHDF5_ENABLE_PARALLEL:BOOL=ON \
     -DHDF5_ENABLE_NONSTANDARD_FEATURE_FLOAT16:BOOL=OFF \
     -DCMAKE_ANSI_CFLAGS:STRING=-fPIC \
     -DCMAKE_ANSI_FCFLAGS:STRING=-fPIC

$HDF5_PATH is a variable in my script. I am using OpenMPI (5.0.7) built with GCC (14.2.0).

With the above options, the HDF5 building seems to work smoothly, but when I try to build my program linking against this HDF5 library, I get errors in the linking phase about undefined references like the following:

/usr/bin/ld: ./lib/libhdf5_fortran.a(H5_ff.F90.o): in function `__h5lib_MOD_h5get_free_list_sizes_f':
H5_ff.F90:(.text+0x4d4): undefined reference to `H5get_free_list_sizes'
...

My program links directly to ./lib/libhdf5_fortran.a (passing also -lz -ldl -lm), but it seems that the static library does not contain all the compiled objects necessary, e.g. H5Aff.o is one of the objects missing. I have also tried to pass all the .a libraries (libhdf5.a, libhdf5_hl.a, ecc…), but the undefined references persist.

On the contrary, building also the shared form of the HDF5 (changing DBUILD_SHARED_LIBS:BOOL=OFF option to ON) and using it to build my program works perfectly (in the linking phase, pass -lhdf5_fortran is enough).

In both shared and static cases, the path of the compiled HDF5 library is in my paths (PATH, LD_LIBRARY_PATH and LD_RUN_PATH).

I am almost sure that I am making some errors in building the static form, and it results in being incomplete, with some compiled objects missing.

Any suggestions are much more than welcome, thank you in advance.

Stefano

2

You don’t mention which version, but you do need to include the correct Fortran mod directory with your build of your target. CMake will add the correct dependencies if you use it for building your target using the find_package(hdf5) command. See the generated hdf5-targets.cmake file for details.
Otherwise you need to simulate the add_library/set_properties details to your target build.
Allen

3

Dear Allen,

Thank you for your quick reply.

You don’t mention which version

I am sorry, I am using the latest release, 1.14.6.

you do need to include the correct Fortran mod directory with your build of your target

I am somehow confused here. Are you referring to the mod directory where fortran .mod files are placed? I.e. the following:

ls ./lib/mod/static
h5a.mod h5ds.mod h5f.mod h5_gen.mod h5im.mod h5l.mod ...

In any case, compiling my program with HDF5 static version works well, I include the `./lib/include’ directory that contains both headers .h and .mod, and the compilation is ok. It is at the linking phase where the undefined references arise. I searched for one of the first references missing, and the corresponding .mod/.o is not present in any subdirectories. My understanding is that I am not building all the HDF5 parts (probably I am wrong).

See the generated hdf5-targets.cmake file for details.

I am not sure what I have to check on this file (I am not an expert of CMake, I use it only for HDF5). The fortran static “targets” (my guess) seem to be present, e.g. build/bin/libhdf5_fortran.a)

Otherwise you need to simulate the add_library/set_properties details to your target build.

This is obscure for me, sorry. Can you explain in more detail?

Stefano

4

I know the CMake pretty well, not much help with Fortran, but the h5get_free_list_sizes_f function is in the MODULE H5LIB.
Do you run CPACK to create an installation image? It will create a subdir named CMake that will have the hdf5-*.cmake files in it. The hdf5-targets.cmake file is where you find the following:


# Create imported target hdf5::hdf5_fortran-static
add_library(hdf5::hdf5_fortran-static STATIC IMPORTED)

set_target_properties(hdf5::hdf5_fortran-static PROPERTIES
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/mod/static;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "hdf5::hdf5_f90cstub-static;\$<LINK_ONLY:m>;\$<LINK_ONLY:dl>;\$<LINK_ONLY:\$<\$<BOOL:OFF>:>>"
)

If you are only compiling one file you might try the h5fc script that should have the correct libs/includes.

Using just the build folder (without running cpack to create an install) for compiling has some extra steps necessary to get everything right. I never do that myself, prefering to use the install image.

5

Do you run CPACK to create an installation image? It will create a subdir named CMake that will have the hdf5-*.cmake files in it

Yes, I have that subdir into my built directories structure. My hdf5-targets.cmake contains the following data appended at the end of this post.

If you are only compiling one file you might try the h5fc script that should have the correct libs/includes

Unfortunately, my program is quite complex and not composed by only one file, but I can try to create e minimal example raising my issue.

Using just the build folder (without running cpack to create an install) for compiling has some extra steps necessary to get everything right. I never do that myself, prefering to use the install image.

Sorry, probably I was not clear. I do also the cmake --build . and make install steps, I have not reported above just because I was thinking that my error is inside the options I passed to the first cmake step.

Contents of my hdf5-targets.cmake

# Generated by CMake

if("${CMAKE_MAJOR_VERSION}.${CMAKE_MINOR_VERSION}" LESS 2.8)
   message(FATAL_ERROR "CMake >= 2.8.0 required")
endif()
if(CMAKE_VERSION VERSION_LESS "2.8.3")
   message(FATAL_ERROR "CMake >= 2.8.3 required")
endif()
cmake_policy(PUSH)
cmake_policy(VERSION 2.8.3...3.26)
#----------------------------------------------------------------
# Generated CMake target import file.
#----------------------------------------------------------------

# Commands may need to know the format version.
set(CMAKE_IMPORT_FILE_VERSION 1)

# Protect against multiple inclusion, which would fail when already imported targets are added once more.
set(_cmake_targets_defined "")
set(_cmake_targets_not_defined "")
set(_cmake_expected_targets "")
foreach(_cmake_expected_target IN ITEMS hdf5-static hdf5-shared mirror_server mirror_server_stop hdf5_tools-static hdf5_tools-shared h5diff ph5diff h5ls h5debug h5repart h5mkgrp h5clear h5delete h5import h5repack h5jam h5unjam h5copy h5stat h5dump h5format_convert h5perf_serial h5perf hdf5_hl-static hdf5_hl-shared h5watch hdf5_f90cstub-static hdf5_f90cstub-shared hdf5_fortran-static hdf5_fortran-shared hdf5_hl_f90cstub-static hdf5_hl_f90cstub-shared hdf5_hl_fortran-static hdf5_hl_fortran-shared)
  list(APPEND _cmake_expected_targets "${_cmake_expected_target}")
  if(TARGET "${_cmake_expected_target}")
    list(APPEND _cmake_targets_defined "${_cmake_expected_target}")
  else()
    list(APPEND _cmake_targets_not_defined "${_cmake_expected_target}")
  endif()
endforeach()
unset(_cmake_expected_target)
if(_cmake_targets_defined STREQUAL _cmake_expected_targets)
  unset(_cmake_targets_defined)
  unset(_cmake_targets_not_defined)
  unset(_cmake_expected_targets)
  unset(CMAKE_IMPORT_FILE_VERSION)
  cmake_policy(POP)
  return()
endif()
if(NOT _cmake_targets_defined STREQUAL "")
  string(REPLACE ";" ", " _cmake_targets_defined_text "${_cmake_targets_defined}")
  string(REPLACE ";" ", " _cmake_targets_not_defined_text "${_cmake_targets_not_defined}")
  message(FATAL_ERROR "Some (but not all) targets in this export set were already defined.\nTargets Defined: ${_cmake_targets_defined_text}\nTargets not yet defined: ${_cmake_targets_not_defined_text}\n")
endif()
unset(_cmake_targets_defined)
unset(_cmake_targets_not_defined)
unset(_cmake_expected_targets)


# Compute the installation prefix relative to this file.
get_filename_component(_IMPORT_PREFIX "${CMAKE_CURRENT_LIST_FILE}" PATH)
get_filename_component(_IMPORT_PREFIX "${_IMPORT_PREFIX}" PATH)
if(_IMPORT_PREFIX STREQUAL "/")
  set(_IMPORT_PREFIX "")
endif()

# Create imported target hdf5-static
add_library(hdf5-static STATIC IMPORTED)

set_target_properties(hdf5-static PROPERTIES
  INTERFACE_COMPILE_DEFINITIONS "_POSIX_C_SOURCE=200809L;_GNU_SOURCE;_LARGEFILE_SOURCE;_FILE_OFFSET_BITS=64"
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "\$<LINK_ONLY:m>;\$<LINK_ONLY:dl>;\$<LINK_ONLY:ZLIB::ZLIB>;\$<\$<PLATFORM_ID:Windows>:>;\$<\$<NOT:\$<PLATFORM_ID:Windows>>:dl>;\$<\$<BOOL:ON>:MPI::MPI_C>;\$<LINK_ONLY:\$<\$<OR:\$<BOOL:OFF>,\$<BOOL:OFF>>:Threads::Threads>>"
)

# Create imported target hdf5-shared
add_library(hdf5-shared SHARED IMPORTED)

set_target_properties(hdf5-shared PROPERTIES
  INTERFACE_COMPILE_DEFINITIONS "H5_BUILT_AS_DYNAMIC_LIB;_POSIX_C_SOURCE=200809L;_GNU_SOURCE;_LARGEFILE_SOURCE;_FILE_OFFSET_BITS=64"
  INTERFACE_INCLUDE_DIRECTORIES "\$<\$<BOOL:OFF>:>;${_IMPORT_PREFIX}/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "\$<\$<PLATFORM_ID:Windows>:>;\$<\$<NOT:\$<PLATFORM_ID:Windows>>:dl>;\$<\$<BOOL:ON>:MPI::MPI_C>;\$<\$<OR:\$<BOOL:>,\$<BOOL:OFF>>:Threads::Threads>"
)

# Create imported target mirror_server
add_executable(mirror_server IMPORTED)

# Create imported target mirror_server_stop
add_executable(mirror_server_stop IMPORTED)

# Create imported target hdf5_tools-static
add_library(hdf5_tools-static STATIC IMPORTED)

set_target_properties(hdf5_tools-static PROPERTIES
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "hdf5-static;\$<LINK_ONLY:\$<\$<BOOL:ON>:MPI::MPI_C>>"
)

# Create imported target hdf5_tools-shared
add_library(hdf5_tools-shared SHARED IMPORTED)

set_target_properties(hdf5_tools-shared PROPERTIES
  INTERFACE_COMPILE_DEFINITIONS "H5_BUILT_AS_DYNAMIC_LIB"
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "hdf5-shared"
)

# Create imported target h5diff
add_executable(h5diff IMPORTED)

# Create imported target ph5diff
add_executable(ph5diff IMPORTED)

# Create imported target h5ls
add_executable(h5ls IMPORTED)

# Create imported target h5debug
add_executable(h5debug IMPORTED)

# Create imported target h5repart
add_executable(h5repart IMPORTED)

# Create imported target h5mkgrp
add_executable(h5mkgrp IMPORTED)

# Create imported target h5clear
add_executable(h5clear IMPORTED)

# Create imported target h5delete
add_executable(h5delete IMPORTED)

# Create imported target h5import
add_executable(h5import IMPORTED)

# Create imported target h5repack
add_executable(h5repack IMPORTED)

# Create imported target h5jam
add_executable(h5jam IMPORTED)

# Create imported target h5unjam
add_executable(h5unjam IMPORTED)

# Create imported target h5copy
add_executable(h5copy IMPORTED)

# Create imported target h5stat
add_executable(h5stat IMPORTED)

# Create imported target h5dump
add_executable(h5dump IMPORTED)

# Create imported target h5format_convert
add_executable(h5format_convert IMPORTED)

# Create imported target h5perf_serial
add_executable(h5perf_serial IMPORTED)

# Create imported target h5perf
add_executable(h5perf IMPORTED)

# Create imported target hdf5_hl-static
add_library(hdf5_hl-static STATIC IMPORTED)

set_target_properties(hdf5_hl-static PROPERTIES
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "hdf5-static"
)

# Create imported target hdf5_hl-shared
add_library(hdf5_hl-shared SHARED IMPORTED)

set_target_properties(hdf5_hl-shared PROPERTIES
  INTERFACE_COMPILE_DEFINITIONS "H5_BUILT_AS_DYNAMIC_LIB"
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "hdf5-shared"
)

# Create imported target h5watch
add_executable(h5watch IMPORTED)

# Create imported target hdf5_f90cstub-static
add_library(hdf5_f90cstub-static STATIC IMPORTED)

set_target_properties(hdf5_f90cstub-static PROPERTIES
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "hdf5-static"
)

# Create imported target hdf5_f90cstub-shared
add_library(hdf5_f90cstub-shared SHARED IMPORTED)

set_target_properties(hdf5_f90cstub-shared PROPERTIES
  INTERFACE_COMPILE_DEFINITIONS "H5_BUILT_AS_DYNAMIC_LIB"
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "hdf5-shared"
)

# Create imported target hdf5_fortran-static
add_library(hdf5_fortran-static STATIC IMPORTED)

set_target_properties(hdf5_fortran-static PROPERTIES
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/mod/static;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "hdf5_f90cstub-static;\$<LINK_ONLY:m>;\$<LINK_ONLY:dl>;\$<LINK_ONLY:\$<\$<BOOL:ON>:>>"
)

# Create imported target hdf5_fortran-shared
add_library(hdf5_fortran-shared SHARED IMPORTED)

set_target_properties(hdf5_fortran-shared PROPERTIES
  INTERFACE_COMPILE_DEFINITIONS "H5_BUILT_AS_DYNAMIC_LIB"
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/mod/shared;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "hdf5_f90cstub-shared"
)

# Create imported target hdf5_hl_f90cstub-static
add_library(hdf5_hl_f90cstub-static STATIC IMPORTED)

set_target_properties(hdf5_hl_f90cstub-static PROPERTIES
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "hdf5_f90cstub-static;hdf5_hl-static"
)

# Create imported target hdf5_hl_f90cstub-shared
add_library(hdf5_hl_f90cstub-shared SHARED IMPORTED)

set_target_properties(hdf5_hl_f90cstub-shared PROPERTIES
  INTERFACE_COMPILE_DEFINITIONS "H5_BUILT_AS_DYNAMIC_LIB"
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/include;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "hdf5_f90cstub-shared;hdf5_hl-shared"
)

# Create imported target hdf5_hl_fortran-static
add_library(hdf5_hl_fortran-static STATIC IMPORTED)

set_target_properties(hdf5_hl_fortran-static PROPERTIES
  INTERFACE_COMPILE_DEFINITIONS "\$<\$<STREQUAL:x,xMSVC>:HDF5F90_WINDOWS>"
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/mod/static;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "hdf5_hl_f90cstub-static;hdf5_fortran-static"
)

# Create imported target hdf5_hl_fortran-shared
add_library(hdf5_hl_fortran-shared SHARED IMPORTED)

set_target_properties(hdf5_hl_fortran-shared PROPERTIES
  INTERFACE_COMPILE_DEFINITIONS "H5_BUILT_AS_DYNAMIC_LIB"
  INTERFACE_INCLUDE_DIRECTORIES "${_IMPORT_PREFIX}/mod/shared;${_IMPORT_PREFIX}/include"
  INTERFACE_LINK_LIBRARIES "hdf5_hl_f90cstub-shared;hdf5_fortran-shared"
)

if(CMAKE_VERSION VERSION_LESS 2.8.12)
  message(FATAL_ERROR "This file relies on consumers using CMake 2.8.12 or greater.")
endif()

# Load information for each installed configuration.
file(GLOB _cmake_config_files "${CMAKE_CURRENT_LIST_DIR}/hdf5-targets-*.cmake")
foreach(_cmake_config_file IN LISTS _cmake_config_files)
  include("${_cmake_config_file}")
endforeach()
unset(_cmake_config_file)
unset(_cmake_config_files)

# Cleanup temporary variables.
set(_IMPORT_PREFIX)

# Loop over all imported files and verify that they actually exist
foreach(_cmake_target IN LISTS _cmake_import_check_targets)
  foreach(_cmake_file IN LISTS "_cmake_import_check_files_for_${_cmake_target}")
    if(NOT EXISTS "${_cmake_file}")
      message(FATAL_ERROR "The imported target \"${_cmake_target}\" references the file
   \"${_cmake_file}\"
but this file does not exist.  Possible reasons include:
* The file was deleted, renamed, or moved to another location.
* An install or uninstall procedure did not complete successfully.
* The installation package was faulty and contained
   \"${CMAKE_CURRENT_LIST_FILE}\"
but not all the files it references.
")
    endif()
  endforeach()
  unset(_cmake_file)
  unset("_cmake_import_check_files_for_${_cmake_target}")
endforeach()
unset(_cmake_target)
unset(_cmake_import_check_targets)

# This file does not depend on other imported targets which have
# been exported from the same project but in a separate export set.

# Commands beyond this point should not need to know the version.
set(CMAKE_IMPORT_FILE_VERSION)
cmake_policy(POP)
6

Thanks for all that info, I try not to assume anything. Unfortunately, your problem might beyond what I can do because it seems that everything to do with CMake is correct. I do know that static libs needs all the dependent libs on the link line. So hdf5_fortran.lib needs hdf5_fortran-cstub which needs hdf5.lib, etc.

Allen

7

Thank you very much Allen,

Excluding that I have not made big mistakes with cmake helps me.

I’ll try to figure out where my error is in other ways.

Stefano

8

Please ask any other questions as you run across them and I will do my best - Fortran specific questions will need one of the Fortran devs to answer.

1 Like
9

Try this simple diagnostic with “nm” to examine the relationship of a program symbol between libraries. U = “Undefined” = library requests an external symbol. T = symbol is provided in this library. On my Mac system (in C shell):

> nm libhdf5_fortran.a |& grep H5get_free_list_sizes$
                 U _H5get_free_list_sizes
> nm libhdf5.a |& grep H5get_free_list_sizes$
0000000000000a0c T _H5get_free_list_sizes

The reported symbols must match exactly for correct linking, including letter case, and leading or trailing underscores. If you find exact match, the libraries are probably built correctly, and the problem is somewhere in your link command. If incorrect match, that is probably a library build problem.

Do I remember correctly, that order of libraries on the linker command line is important for static linking?

10

Do I remember correctly, that order of libraries on the linker command line is important for static linking?

Dear Dave, I think you are right, but I have to admit that I have tested only one order (probably the wrong one) and now I am out of office. Tomorrow I will perform the nm test and the other order of linking (I almost sure this is my error). Thank you very much.

Stefano

11

Dear Dave,

Thank you very much for pointing out the nm. It has been fundamental to find the correct order of the linked libraries. I have finally built my program against the static form of HDF5. I had passed all the .a in my previous test, but in the wrong order, using nm I found the correct one.

If other will face with my error, for my program the correct order is the following:

libhdf5_fortran.a libhdf5_f90cstub.a libhdf5.a

Thank you again to all.

Stefano