Building with zlib (and szip) on windows

HDF5 Library C++ Users' Group
1

I’m trying to build hdf5 with sz and zlib support, but I got very confused; I can’t figure out an easy way how to.
My main question probably is - can somebody point me to a tutorial or writeup where it is described how to easily build hdf5 with those two included on Windows? Preferrably from the CMake sources, since this is what I build on previously and have at least some familiarity with.

As a little context - for previous versions (1.10 and probably 1.12) hfd5 was built “out of the box” with sz and zlib support for me (when using the CMake sources). But shis seems to have stopped somehow. I can’t figure out how to configure the build such that it will build those libraries. There seem to be a variety of options of how to include them, with many involved settings in CMake, among them

  • HDF5_ALLOW_EXTERNAL_SUPPORT
  • HDF5_ENABLE_SZIP_SUPPORT
  • HDF5_ENABLE_Z_LIB_SUPPORT

When I try to CMake-configure the build folder “manually”, those last two keep disabling themselves automatically, with no hint why, no matter what other settings I configure, except for the HDF5_ALLOW_EXTERNAL_SUPPORT=NO case, where however I cannot figure out what tio set the SZIP_DIR nor ZLIB_DIR to - do these need to be separate builds of those libraries? Can hdf5 not automatically build these libraries within its build anymore (as I’m pretty sure it used to do for me at a previous point).
The CMake sources stilll do include the zlib sources (and libaec, which I think has something to do with sz, right?), but I can’t figure out why they are not built or used (or what I need to do to make them used).

My basic current workflow is downloading the CMake sources from Index of /ftp/HDF5/releases/hdf5-1.14/hdf5-1.14.3/src, adapting the HDF5options.cmake file (basically, disabling shared_libs build and disabling testing, examples, tools and utils since I really only need the static libraries).

Sorry if this is very basic stuff that is properly documented somewhere, I just really can’t find it / figure it out at the moment!

2

Have you tried vcpkg/ports/hdf5 at master · microsoft/vcpkg · GitHub?
This will be probably the easiest way to install h5 w/ sz+zlib for Windows users.

P.S. You can try my fork for 1.14.4 if you need the latest h5:

3

We are currently not using vcpkg, incorporating it into our build process just for hdf5 feels a bit overkill, but I might look into it in the future. I’m curently testing just using the binary release for now.

4

Can you get the source from github (1.14 has had three releases now) instead of ftp? Then we know we are using the same source files. (Just to eliminate any variables there)
What version of CMake are you using, CMake 3.25+ will allow use of CMakePresets which is nice because we could create a UserPresets file just for your needs.

Compression filters do have a number of options for flexibility, however I prefer to build them in-line on Windows. Even then there are options depending on if you want to download them from the source repos or use a local tgz file. The latest release has the best CMake process for this but even the earlier 1.14 releases should work well.

The HDOptions.cmake / CTestScript.cmake process we were supplying should use the config/cmake/cacheinit.cmake file and it will set most all of the options that follow.
The basic settings for using the filters from a local folder are:
-DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING=TGZ
-DTGZPATH:PATH=${CTEST_SCRIPT_DIRECTORY} # this is where the build script is run from and you may need to chage this if you use the command line instead of the HDFOptions file
-DHDF5_ENABLE_SZIP_SUPPORT:BOOL=ON
-DHDF5_ENABLE_SZIP_ENCODING:BOOL=ON
-DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=ON
-DLIBAEC_TGZ_NAME:STRING=libaec-1.0.6.tar.gz or libaec-v1.1.3.tar.gz #match the filename to your version
-DZLIB_TGZ_NAME:STRING=zlib.1.3.tar.gz #match the filename to your version
-DLIBAEC_USE_LOCALCONTENT:BOOL=ON
-DZLIB_USE_LOCALCONTENT:BOOL=ON

Using the HDFOptions.cmake file you just need to add/edit these options

1 Like
5

FYI - I updated my fork to use 1.14.4.3 which was released yesterday:

c:\Users\JoeLee\source\repos\vcpkg>.\vcpkg install hdf5
.\vcpkg install hdf5
Computing installation plan...
The following packages will be built and installed:
    hdf5[core,szip,zlib]:x64-windows@1.14.4.3
Detecting compiler hash for triplet x64-windows...
Compiler found: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe
Restored 0 package(s) from C:\Users\JoeLee\AppData\Local\vcpkg\archives in 81.8 us. Use --debug to see more details.
Installing 1/1 hdf5[core,szip,zlib]:x64-windows@1.14.4.3...
Building hdf5[core,szip,zlib]:x64-windows@1.14.4.3...
-- Downloading https://github.com/HDFGroup/hdf5/archive/hdf5_1.14.4.3.tar.gz -> HDFGroup-hdf5-hdf5_1.14.4.3.tar.gz...
-- Extracting source C:/Users/JoeLee/source/repos/vcpkg/downloads/HDFGroup-hdf5-hdf5_1.14.4.3.tar.gz
-- Applying patch hdf5_config.patch
-- Applying patch szip.patch
-- Using source at C:/Users/JoeLee/source/repos/vcpkg/buildtrees/hdf5/src/5_1.14.4.3-3e0512f185.clean
-- Found external ninja('1.11.0').
-- Configuring x64-windows-dbg
-- Configuring x64-windows-rel
-- Building x64-windows-dbg
-- Building x64-windows-rel
-- Fixing pkgconfig file: C:/Users/JoeLee/source/repos/vcpkg/packages/hdf5_x64-windows/lib/pkgconfig/hdf5.pc
-- Fixing pkgconfig file: C:/Users/JoeLee/source/repos/vcpkg/packages/hdf5_x64-windows/lib/pkgconfig/hdf5_hl.pc
-- Using cached msys2-mingw-w64-x86_64-pkgconf-1~2.2.0-1-any.pkg.tar.zst.
-- Using cached msys2-msys2-runtime-3.4.10-4-x86_64.pkg.tar.zst.
-- Using msys root at C:/Users/JoeLee/source/repos/vcpkg/downloads/tools/msys2/d33df895d441a36d
-- Fixing pkgconfig file: C:/Users/JoeLee/source/repos/vcpkg/packages/hdf5_x64-windows/debug/lib/pkgconfig/hdf5.pc
-- Fixing pkgconfig file: C:/Users/JoeLee/source/repos/vcpkg/packages/hdf5_x64-windows/debug/lib/pkgconfig/hdf5_hl.pc
-- Performing post-build validation
Stored binaries in 1 destinations in 2.2 s.
Elapsed time to handle hdf5:x64-windows: 1.7 min
hdf5:x64-windows package ABI: 26b309c6d1cbaf72ed5dd6ecb2cfecc4797aba4c76b481f041368e52561a2eaf
Total install time: 1.7 min
hdf5 provides CMake targets:

  # this is heuristically generated, and may not be correct
  find_package(hdf5 CONFIG REQUIRED)
  target_link_libraries(main PRIVATE hdf5::hdf5-shared hdf5::hdf5_hl-shared)

hdf5 provides pkg-config modules:

  # HDF5 (Hierarchical Data Format 5) Software Library
  hdf5

  # HDF5 (Hierarchical Data Format 5) Software Library
  hdf5_hl

I hope it can save time for Windows users.

I also created PR to Microsoft: [hdf5] Update version to 1.14.4.3 by hyoklee · Pull Request #38911 · microsoft/vcpkg (github.com).

1 Like
6

Thanks for t he detailed instructions!

I have now tried building via CMake directly from github source. I also realized that at the moment I really only need zlib support, sz is I think not that important for us.

So I left out the sz/aec-related options, and tried:

cmake -DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING="TGZ" -DTGZPATH:PATH="C:/Tools/hdf5"
    -DBUILD_SHARED_LIBS:BOOL=OFF -DHDF5_BUILD_TOOLS:BOOL=OFF
    -DHDF5_BUILD_UTILS:BOOL=OFF -DBUILD_TESTING:BOOL=OFF
    -DHDF5_BUILD_EXAMPLES:BOOL=OFF -DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=ON
    -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF -DZLIB_TGZ_NAME:STRING=zlib-1.3.1.tar.gz
    -DZLIB_USE_LOCALCONTENT:BOOL=ON ..\src.git

zlib-1.3.1.tar.gz is downloaded directly from Releases · madler/zlib · GitHub

but I am getting an error:

CMake Error at config/cmake/HDFLibMacros.cmake:36 (add_library):
  add_library cannot create ALIAS target "zlib-static" because another target
  with the same name already exists.
Call Stack (most recent call first):
  CMakeFilters.cmake:85 (EXTERNAL_ZLIB_LIBRARY)
  CMakeLists.txt:870 (include)

CMake Warning at CMakeFilters.cmake:107 (message):
   ZLib support in HDF5 was enabled but not found
Call Stack (most recent call first):
  CMakeLists.txt:870 (include)

EDIT:

Using the HDFOptions.cmake file you just need to add/edit these options

That file only exists in the released CMake-hdf5-x.y.z.tar.gz files, right? Basically I was trying to do this in my previous tries (see my first post), but couldn’t get it to work the way I want, and the way I previously remember it working: that it automatically built zlib (and sz) and also provided those dependent libraries along with the exported CMake targets.

I did get a working build by building zlib myself; with HDF5 configured with these options:

-DZLIB_DIR:PATH=C:/Tools/hdf5/zlib/install/cmake (the cmake folder under where I installed the self-built zlib)
-DZLIB_USE_EXTERNAL:BOOL=0
-DBUILD_TESTING:BOOL=0
-DHDF5_BUILD_TOOLS:BOOL=0
-DHDF5_BUILD_EXAMPLES:BOOL=0
-DHDF5_BUILD_UTILS:BOOL=0
-DZLIB_USE_EXTERNAL:BOOL=0
-DBUILD_SHARED_LIBS:BOOL=0

(Note: not sure what ZLIB_USE_EXTERNAL actually means here - I do use an externally built zlib, but I still need to turn it off, otherwise I get a message saying that zlib is turned on but not properly configured…)

Downside then is that in my build I need to manually link to zlib (and also hdf5 doesn’t seem to create the hdf5::hdf5-static alias, I need to link to hdf5-static). So there seems to some automatism in place that puts together a combined hdf5+zlib “release” (at least that was the case before when I was using the CMake-hdf5-x.y.z.tar.gz sources), but that doesn’t seem to happen when building “manually” directly from the hdf5 sources?

7

Def: Use External Library Building for ZLIB else search - which means set to 0 it will search for the package on the system. Set 1 and it will build the zlib package from source (tgz or git and loacl vs download)

8

So a couple things I noticed:
The tgzpath must be a full path and not relative (You do that already, but it is news to me)
Be sure the zlib name is correct - my file was zlib-1.3.tar.gz (no .1)
Add the -D for ZLIB_USE_EXTERNAL:BOOL=ON
You should specify the generator with -G""
Add --log-level=VERBOSE it will tell you the actual problem.
I would also add “-C /config/cmake/cacheinit.cmake” and it can be relative, but you will need to disable a couple of more items; -DHDF5_BUILD_JAVA:BOOL=OFF, -DHDF5_BUILD_FORTRAN:BOOL=OFF, -DHDF5_BUILD_CPP_LIB:BOOL=OFF
If you look at cacheinit.cmake in the source you’ll see what it sets. NOTE that witout using the cacheinit file I got the same error as you. Looks like I need to review the defaults vs that file.

Allen

1 Like
9

Thanks for these insights!
I could finally build hdf5 with zlib again, with this cmake configuration:
(in directory C:\Tools\hdf5\bin, with git sources in C:\Tools\hdf5\src.git, at git tag hdf5_1.14.4.3):

cmake -G "Visual Studio 17 2022" -C ../src.git/config/cmake/cacheinit.cmake
    --log-level=VERBOSE
    -DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING="TGZ" -DTGZPATH:PATH="C:/Tools/hdf5"
    -DBUILD_SHARED_LIBS:BOOL=OFF -DHDF5_BUILD_TOOLS:BOOL=OFF
    -DHDF5_BUILD_UTILS:BOOL=OFF -DBUILD_TESTING:BOOL=OFF
    -DHDF5_BUILD_EXAMPLES:BOOL=OFF -DHDF5_BUILD_JAVA:BOOL=OFF
	-DHDF5_BUILD_FORTRAN:BOOL=OFF -DHDF5_BUILD_CPP_LIB:BOOL=OFF
	-DHDF5_ENABLE_Z_LIB_SUPPORT:BOOL=ON -DZLIB_TGZ_NAME:STRING=zlib-1.3.1.tar.gz
    -DZLIB_USE_LOCALCONTENT:BOOL=ON -DZLIB_USE_EXTERNAL:BOOL=ON
    -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=OFF ..\src.git

If I have time I’ll look into the difference between this build and my previous tries with the CMake-hdf5-x.y.z.tar.gz sources…

On a related note, is it planned for these CMake-… bundles to be be continued? I couldn’t find one for 1.14.4 on the release ftp.

10

The build works fine and zlib is built together with hdf5. Now I’m however running into another problem: CMake doesn’t accept the hdf5 package created with this configuration - it reports a missing target hdf5::zlib-static. As far as I can tell, I have done the build and install in the “standard” way (Visual Studio - Build solution, and then “building” the INSTALL target).

My program does

find_package(HDF5 NAMES hdf5 COMPONENTS C static NO_MODULE QUIET)
if (HDF5_FOUND)
	message(STATUS "HDF5: ${HDF5_VERSION} in ${HDF5_DIR}")
else()
	message(STATUS "HDF5: Not found - ${HDF5_NOT_FOUND_MESSAGE}")
endif()

(which has worked nicely for me up until now with all other hdf5 builds I did).

cmake output:

HDF5: Not found - The following imported targets are referenced, but are missing:  hdf5::zlib-static

Took me a while to figure out getting information about this, didn’t know about the ...NOT_FOUND_MESSAGE so far…

Adding the --debug-find option to cmake did not give me additional insights, there is no mention of hdf5::zlib-static in its output.
The zlib-*.cmake files exist in the cmake folder of the install location, and they are referenced from hdf5-config as far as I can tell…

Any idea how to fix this?

11

One thing we did in the old process is set the following option ON:
option (HDF5_PACKAGE_EXTLIBS “CPACK - include external libraries” OFF)
So just add a -DHDF5_PACKAGE_EXTLIBS:BOOL=ON

Allen

12

That option seems to have done it, now it works. Thanks!

13

I wonder if we should change that default?