Problems building HDF5 with plugin support

Some preliminaries:

  • Compiling from source for hdf5-1.14.6 (also tried hdf5-1.14.5)
  • Compiling on Linux with Intel compilers 2023.2.1 (icx and ifort)
  • Using CMake, as Autotools option doesn’t seem to support hdf5_plugins
  • Using “Unix Makefiles”

Between INSTALL_CMake.txt and discussion in Compiling HDF5 with Plugins (LZF), it appears HDF5 and hdf5_filters can be build simultaneously using the HDF5 cmake invocation. However, cmake fails in the first step. My invocation:

cmake -G “Unix Makefiles”
-DCMAKE_BUILD_TYPE:STRING=Release
-DBUILD_SHARED_LIBS:BOOL=OFF
-DBUILD_TESTING:BOOL=ON
-DHDF5_BUILD_FORTRAN:BOOL=ON
-DHDF5_INSTALL_MOD_FORTRAN:BOOL=ON
-DHDF5_BUILD_TOOLS:BOOL=ON
-DCMAKE_C_FLAGS:STRING=“-fpic”
-DHDF5_ENABLE_PLUGIN_SUPPORT:BOOL=ON
-DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING=“TGZ”
-DPLUGIN_TGZ_NAME:STRING=“hdf5_plugins-1.14.tar.gz”
-DTGZPATH:PATH=“/discover/nobackup/projects/usaf_lis/emkemp/ZSTD/intel-2023.2.1/tarballs”
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF
…/hdf5-1.14.6 >& config.log

The error message is:
[ 22%] Performing download step (DIR copy) for ‘hdf5_zlib-populate’
Error copying directory from “/” to “/discover/nobackup/projects/usaf_lis/emkemp
/ZSTD/intel-2023.2.1/myhdfstuff/build/_deps/hdf5_zlib-src”.
gmake[2]: *** [CMakeFiles/hdf5_zlib-populate.dir/build.make:100: hdf5_zlib-popul
ate-prefix/src/hdf5_zlib-populate-stamp/hdf5_zlib-populate-download] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/hdf5_zlib-populate.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2

Modifying the -DTGZPATH flag to replace :PATH with :STRING does not change this behavior.

I assume an additional flag is required to specify the source directory (instead of "/'), but I don’t know what this is.

Any suggestions?

Hi, @eric.kemp !

You may try something like these:
hdf5_plugins/.github/workflows/cmake-ctest.yml at master · HDFGroup/hdf5_plugins
hdf5_plugins/.github/workflows/cmake-script.yml at master · HDFGroup/hdf5_plugins

However, I cannot guarantee that it will work based on the current CI results:
Workflow runs · HDFGroup/hdf5_plugins

This is the library compression zlib, not a plugin. Because you are using TGZ option the zlib tgz file must also be at that location.

Thanks @byrn. I made several attempts to modify the cmake command line and got different errors. For simplicity, the below examples don’t try to build the hdf5_plugins.

Attempt 1, mimicking INSTALL_CMake.txt IV-2-C:

cmake -G “Unix Makefiles”
-DCMAKE_BUILD_TYPE:STRING=Release
-DBUILD_SHARED_LIBS:BOOL=OFF
-DBUILD_TESTING:BOOL=ON
-DHDF5_BUILD_FORTRAN:BOOL=ON
-DHDF5_INSTALL_MOD_FORTRAN:BOOL=ON
-DHDF5_BUILD_TOOLS:BOOL=ON
-DCMAKE_C_FLAGS:STRING=“-fpic”
-DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING=“TGZ”
-DZLIB_TGZ_NAME:STRING=“zlib-1.3.1.tar.gz”
-DTGZPATH:STRING=“/discover/nobackup/projects/usaf_lis/emkemp/ZSTD/intel-2023.2.1/tarballs/”
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF
…/hdf5-1.14.6 >& config.log

Error message:
[ 11%] Creating directories for ‘hdf5_zlib-populate’
[ 22%] Performing download step (verify and extract) for ‘hdf5_zlib-populate’
CMake Error at hdf5_zlib-subbuild/hdf5_zlib-populate-prefix/src/hdf5_zlib-populate-stamp/verify-hdf5_zlib-populate.cmake:11 (message):
File not found: /zlib-1.3.1.tar.gz

gmake[2]: *** [CMakeFiles/hdf5_zlib-populate.dir/build.make:99: hdf5_zlib-populate-prefix/src/hdf5_zlib-populate-stamp/hdf5_zlib-populate-download] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/hdf5_zlib-populate.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2

CMake Error at /usr/share/cmake/Modules/FetchContent.cmake:1012 (message):
Build step for hdf5_zlib failed: 2
Call Stack (most recent call first):
/usr/share/cmake/Modules/FetchContent.cmake:1141:EVAL:2 (__FetchContent_directPopulate)
/usr/share/cmake/Modules/FetchContent.cmake:1141 (cmake_language)
/usr/share/cmake/Modules/FetchContent.cmake:1184 (FetchContent_Populate)
config/cmake/HDFLibMacros.cmake:38 (FetchContent_MakeAvailable)
CMakeFilters.cmake:140 (EXTERNAL_ZLIB_LIBRARY)
CMakeLists.txt:921 (include)

Attempt 2, mimicking INSTALL_CMake.txt IV-2-D:
cmake -G “Unix Makefiles”
-DCMAKE_BUILD_TYPE:STRING=Release
-DBUILD_SHARED_LIBS:BOOL=OFF
-DBUILD_TESTING:BOOL=ON
-DHDF5_BUILD_FORTRAN:BOOL=ON
-DHDF5_INSTALL_MOD_FORTRAN:BOOL=ON
-DHDF5_BUILD_TOOLS:BOOL=ON
-DCMAKE_C_FLAGS:STRING=“-fpic”
-DZLIB_TGZ_NAME:STRING=“zlib-1.3.1.tar.gz”
-DZLIB_TGZ_ORIGPATH:STRING=“/discover/nobackup/projects/usaf_lis/emkemp/ZSTD/intel-2023.2.1/tarballs/”
-DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING=“TGZ”
-DZLIB_USE_LOCALCONTENT:BOOL=OFF
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF
…/hdf5-1.14.6 >& config.log

Error message:
– Renaming
– /discover/nobackup/projects/usaf_lis/emkemp/ZSTD/intel-2023.2.1/myhdfstuff/build/_deps/hdf5_zlib-src/zconf.h
– to ‘zconf.h.included’ because this file is included with zlib
– but CMake generates it automatically in the build directory.
CMake Error at config/cmake/HDFLibMacros.cmake:40 (add_library):
add_library cannot create ALIAS target “zlib-static” because another target
with the same name already exists.
Call Stack (most recent call first):
CMakeFilters.cmake:140 (EXTERNAL_ZLIB_LIBRARY)
CMakeLists.txt:921 (include)

CMake Warning at CMakeFilters.cmake:163 (message):
ZLib support in HDF5 was enabled but not found
Call Stack (most recent call first):
CMakeLists.txt:921 (include)

[…snip…]

– HDF5 H5_LIBVER_DIR: 114 HDF5_VERSION_MAJOR: 1.14
– Configuring incomplete, errors occurred!

Attempt 3 follows the very end of mimicking INSTALL_CMake.txt IV-2-D, from “The files can also be retrieved from a local path if necessary”

cmake -G “Unix Makefiles”
-DCMAKE_BUILD_TYPE:STRING=Release
-DBUILD_SHARED_LIBS:BOOL=OFF
-DBUILD_TESTING:BOOL=ON
-DHDF5_BUILD_FORTRAN:BOOL=ON
-DHDF5_INSTALL_MOD_FORTRAN:BOOL=ON
-DHDF5_BUILD_TOOLS:BOOL=ON
-DCMAKE_C_FLAGS:STRING=“-fpic”
-DZLIB_TGZ_NAME:STRING=“zlib-1.3.1.tar.gz”
-DTGZPATH:STRING=“/discover/nobackup/projects/usaf_lis/emkemp/ZSTD/intel-2
023.2.1/tarballs/”
-DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING=“TGZ”
-DZLIB_USE_LOCALCONTENT:BOOL=ON
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF
…/hdf5-1.14.6 >& config.log

But I get the same error.

Update: I reviewed available CMake files and found a combination that builds HDF5 with ZLIB (and SZIP, for simplicity in running the automated tests). The selected arguments are:

cmake -G “Unix Makefiles”
-C …/hdf5-1.14.6/config/cmake/cacheinit.cmake
-DCMAKE_BUILD_TYPE:STRING=Release
-DBUILD_SHARED_LIBS:BOOL=OFF
-DCMAKE_C_FLAGS:STRING=“-fpic”
-DHDF5_BUILD_FORTRAN:BOOL=ON
-DHDF5_INSTALL_MOD_FORTRAN:BOOL=ON
-DHDF5_BUILD_JAVA:BOOL=OFF
-DCMAKE_INSTALL_PREFIX:PATH=/discover/nobackup/projects/usaf_lis/emkemp/ZSTD/intel-2023.2.1/installs/hdf5-1.14.6
-DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING=TGZ
-DTGZPATH:PATH=/discover/nobackup/projects/usaf_lis/emkemp/ZSTD/intel-2023.2.1/myhdfstuff
-DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=ON
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=ON
-DHDF5_ENABLE_SZIP_ENCODING:BOOL=ON
-DHDF5_BUILD_CPP_LIB:BOOL=OFF
-DHDF5_BUILD_FORTRAN:BOOL=ON
-DHDF5_BUILD_HL_LIB:BOOL=OFF
-DBUILD_TESTING:BOOL=ON
-DHDF5_PACKAGE_EXTLIBS:BOOL=ON
/discover/nobackup/projects/usaf_lis/emkemp/ZSTD/intel-2023.2.1/myhdfstuff/hdf5-1.14.6 >& config.log

So I’m over that hurdle. Will tackle hdf5_plugins tomorrow.

Last (?) post. I can get HDF5 to compile with ZSTD support with the following:

cmake -G “Unix Makefiles”
-C …/hdf5-1.14.6/config/cmake/cacheinit.cmake
-DCMAKE_BUILD_TYPE:STRING=Release
-DBUILD_SHARED_LIBS:BOOL=ON
-DBUILD_STATIC_LIBS:BOOL=OFF
-DHDF5_BUILD_FORTRAN:BOOL=ON
-DHDF5_BUILD_CPP_LIB:BOOL=OFF
-DHDF5_BUILD_HL_LIB:BOOL=OFF
-DHDF5_INSTALL_MOD_FORTRAN:STRING=SHARED
-DHDF5_BUILD_JAVA:BOOL=OFF
-DCMAKE_INSTALL_PREFIX:PATH=/discover/nobackup/projects/usaf_lis/emkemp/ZSTD/intel-2023.2.1/installs/hdf5-1.14.6
-DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING=TGZ
-DTGZPATH:PATH=/discover/nobackup/projects/usaf_lis/emkemp/ZSTD/intel-2023.2.1/myhdfstuff
-DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=ON
-DZLIB_TGZ_NAME:STRING=zlib-1.3.1.tar.gz
-DZLIB_USE_LOCALCONTENT:BOOL=ON
-DHDF5_USE_ZLIB_NG:BOOL=OFF
-DLIBAEC_TGZ_NAME:STRING=libaec-1.1.3.tar.gz
-DLIBAEC_USE_LOCALCONTENT:BOOL=ON
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=ON
-DHDF5_ENABLE_SZIP_ENCODING:BOOL=ON
-DBUILD_TESTING:BOOL=ON
-DHDF5_PACKAGE_EXTLIBS:BOOL=ON
-DHDF5_ENABLE_ALL_WARNINGS:BOOL=ON
-DHDF5_ENABLE_PLUGIN_SUPPORT:BOOL=ON
-DPLUGIN_USE_EXTERNAL:BOOL=ON
-DPLUGIN_TGZ_NAME:STRING=hdf5_plugins-1.14.tar.gz
-DPLUGIN_USE_LOCALCONTENT:BOOL=ON
-DZSTD_TGZ_NAME=zstd-1.5.6.tar.gz
-DH5PL_TGZPATH:PATH=/discover/nobackup/projects/usaf_lis/emkemp/ZSTD/intel-2023.2.1/myhdfstuff
-DENABLE_BLOSC:BOOL=OFF
-DENABLE_BLOSC2:BOOL=OFF
-DENABLE_BSHUF:BOOL=OFF
-DENABLE_BZIP2:BOOL=OFF
-DENABLE_FPZIP:BOOL=OFF
-DENABLE_JPEG:BOOL=OFF
-DENABLE_LZ4:BOOL=OFF
-DENABLE_LZF:BOOL=OFF
-DENABLE_SZ:BOOL=OFF
-DENABLE_ZFP:BOOL=OFF
-DHDF_ZSTD_USE_EXTERNAL:BOOL=ON
-DH5PL_BUILD_TESTING:BOOL=ON
-DH5PL_COMP_TGZPATH:PATH=/discover/nobackup/projects/usaf_lis/emkemp/ZSTD/intel-2023.2.1/myhdfstuff
/discover/nobackup/projects/usaf_lis/emkemp/ZSTD/intel-2023.2.1/myhdfstuff/hdf5-1.14.6

Build, test, and install steps work, but it was necessary to update my LD_LIBRARY_PATH before running ctest.

Yes, we have discovered the same, and a little investigation and discussion provides a too obvious solution when building plugins inline: we know where we are going to install the plugins and we should just set the default path. We will be fixing this in develop but all you need to do is to add the path to the plugins on the command line:
A new configuration option can be used to change the default plugin path:

  • Autotools configure option –with-default-plugindir=location
  • CMake option -DH5_DEFAULT_PLUGINDIR:PATH=location