System installed zlib not being used when building HDF5 2.0 with cmake

HDF5 Library
1

I am working on building hdf5-2.0.0 on a rhel8 system. It currently has zlib-1.2.11 installed. Reading through the instructions in the INSTALL_Cmake.txt file, specifically:

  1. If you plan to use Zlib or Szip (aka libaec):
    A. Download the binary packages and install them in a central location.
    For example on Windows, create a folder extlibs and install the
    packages there. Add the following CMake options:
    -DZLIB_LIBRARY:FILEPATH=some_location/lib/zlib.lib
    -DZLIB_INCLUDE_DIR:PATH=some_location/include
    -DZLIB_USE_EXTERNAL:BOOL=OFF
    -DSZIP_LIBRARY:FILEPATH=some_location/lib/libszaec.lib
    -DSZIP_INCLUDE_DIR:PATH=some_location/include
    -Dlibaec_LIBRARY:FILEPATH=some_location/lib/libaec.lib
    -Dlibaec_INCLUDE_DIR:PATH=some_location/include
    -DSZIP_USE_EXTERNAL:BOOL=OFF
    where “some_location” is the full path to the extlibs folder.
    Also if the appropriate environment variable is set, the above options are not required;
    set(ENV{ZLIB_ROOT} “some_location”)
    set(ENV{SZIP_ROOT} “some_location”)
    set(ENV{libaec_ROOT} “some_location”)

I am under the impression that with zlib already installed on my rhel8 system, I can use that in place of downloading it again. Unfortunately that does not appear to be the case. Below is the cmake command I am using.

cmake -G 'Unix Makefiles' -DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_TESTING:BOOL=ON -DHDF5_BUILD_TOOLS:BOOL=ON -DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=ON -DZLIB_USE_EXTERNAL:BOOL=OFF -DZLIB_USE_LOCALCONTENT:BOOL=ON -DZLIB_LIBRARY:FILEPATH=/usr/lib64/libz.so -DZLIB_INCLUDE_DIR:PATH=/usr/include -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=ON -DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING=TGZ -DLIBAEC_TGZ_NAME:STRING=libaec-1.1.3.tar.gz -DLIBAEC_TGZ_ORIGPATH:STRING=%{_hdf5_build_loc}/libaec/ -DLIBAEC_USE_LOCALCONTENT:BOOL=OFF -DBUILD_SHARED_LIBS:BOOL=ON ../hdf5-2.0.0

This will build without error on my system. It will however state that it did not use the ZLIB_LIBRARY and ZLIB_INCLUDE_DIR. Upon installing this and attempting to use it, I find I am unable to read h5 files with deflate compression:

GROUP “RH::SFC::20250912_1500–20250912_1600” {
DATASET “Data” {
DATATYPE H5T_STD_I8LE
DATASPACE SIMPLE { ( 289, 417 ) / ( H5S_UNLIMITED, H5S_UNLIMITED ) }
DATA {h5dump error: unable to print data

     }
  }

}

I have also written a small python program with h5py to test gzip compression, and it too fails when reading or writing a gzip compressed file.

This can be solved if instead of relying on the system installed zlib, I download and include the .tar.gz file much like libaec. I have the following questions on this issue:

  1. Why I am receiving no errors when this is building, yet zlib support does not seem to be built in or included.
  2. If I am specifying a local libaec to be built, does this in turn require zlib to be built in the same manner?
2

Hi @joseph.kelmer,

I believe the issues you’re running into mostly come down to our documentation needing to be updated, as there has been a fair amount of churn regarding how to make use of compression libraries in HDF5.

For using a system-installed zlib, the only option you should need to enable is -DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=ON, at which point CMake should be able to locate your system-installed zlib. Depending on how your zlib was built, you may also need to specify -DHDF5_MODULE_MODE_ZLIB=ON to influence how CMake locates your zlib installation, but usually this isn’t necessary. Finally, if your zlib is in a non-system path, you may also need to pass -DZLIB_ROOT=/path/to/zlib to point to where zlib is, but it seems that shouldn’t be needed in your case. The other variables you’ve set, such as -DZLIB_USE_LOCALCONTENT:BOOL=ON, mostly relate to how we build these compression libraries from source when building HDF5 binaries and shouldn’t be necessary for most use cases. In fact, I believe some of them are actively preventing you from being able to use your system-installed zlib.

The situation with libaec is a little bit trickier. You should be able to request only libaec to be used from the local source with the szip/libaec variables you already have set (though -DSZIP_USE_EXTERNAL=ON should be added). The ZLIB_USE_EXTERNAL / SZIP_USE_EXTERNAL variables independently control whether zlib and szip/libaec should be built from external sources (ON) or just use a system-installed version (OFF). However, it’s worth noting that the HDF5_ALLOW_EXTERNAL_SUPPORT variable is a bit restrictive in that it controls how sources are obtained for every library that you specify HDF5 should get from external sources. So if you were to specify both zlib and libaec as coming from external sources, they’d both have to either come from a tar.gz file (HDF5_ALLOW_EXTERNAL_SUPPORT=TGZ) or from git repositories (HDF5_ALLOW_EXTERNAL_SUPPORT=GIT).

For HDF5 2.0.0 and earlier, it was fairly difficult to include support for libaec without either building it from source or using a tar.gz file. As of the upcoming HDF5 2.1.0, you should be able to use a system-installed libaec the same as zlib by only specifying -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=ON (and optionally -Dlibaec_ROOT if needed for a non-system path installation of libaec). If possible, I’d recommend that as the preferred approach going forward, though using libaec from a tar.gz file will still be supported. If you’d like that functionality in HDF5 2.0.0, you’d need to backport the changes in Add Findlibaec CMake module for locating libaec (#6152) · HDFGroup/hdf5@b100609 · GitHub and Distribute Findlibaec.cmake module for use by HDF5 library consumers … · HDFGroup/hdf5@9af6db1 · GitHub.

So, for HDF5 2.0.0 a revised build command such as:

cmake -G ‘Unix Makefiles’ -DCMAKE_BUILD_TYPE:STRING=Release -
DBUILD_TESTING:BOOL=ON -DHDF5_BUILD_TOOLS:BOOL=ON -DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=ON -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=ON -DSZIP_USE_EXTERNAL=ON -DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING=TGZ -DLIBAEC_TGZ_NAME:STRING=libaec-1.1.3.tar.gz -DLIBAEC_TGZ_ORIGPATH:STRING=%{_hdf5_build_loc}/libaec/ -DLIBAEC_USE_LOCALCONTENT:BOOL=OFF -DBUILD_SHARED_LIBS:BOOL=ON ../hdf5-2.0.0

should work for using a system-installed zlib and a libaec built from the tar.gz file, but please let me know if you encounter issues with that, as it’s a bit of an edge case mixing the two which we don’t really test currently. For HDF5 2.1.0 and on, it would likely be easiest to install libaec on the system and just use:

cmake -G ‘Unix Makefiles’ -DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_TESTING:BOOL=ON -DHDF5_BUILD_TOOLS:BOOL=ON -DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=ON -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=ON ../hdf5-2.1.0

(assuming that both zlib and libaec are installed system-wide)

3

Hey there, I greatly appreciate you reaching out so quickly. I just tested your suggestion, and unfortunately it is still not using the system ZLIB libraries. My intention is to build the HDF5 packages without reaching out for external sources. With this CMake command, once the configuration finds the stdatomic.h, it will then immediately attempt to download ZLIB from the git repository. That is why I ended up adding those arguments such as the -DZLIB_USE_LOCALCONTENT:BOOL=ON, as those eventually allowed the build to complete without error while not connected to the internet.

If I were connected to the internet, that command would work. It will grab the ZLIB from git and use that, alongside building the libaec from the tar.gz. However, that is not the use case that I am attempting to get working here.

Again, thank you for looking into this and responding. If you happen to have any further suggestions I am all ears.

4

With this CMake command, once the configuration finds the stdatomic.h, it will then immediately attempt to download ZLIB from the git repository.

Just checking, but did you happen to start from a completely fresh and empty build directory? As long as ZLIB_USE_EXTERNAL is unspecified or specified as OFF, the build process shouldn’t ever try and download zlib from git, it should only try to find it on the system.

5

Correct. With a completely fresh build environment it is looking off system for the ZLIB installation, and both with ZLIB_USE_EXTERNAL unspecified and with it set to OFF

6

Hmm maybe we need to do a little more digging to figure out where our builds are differing on this one. For reference I just ran the following steps:

git clone git@github.com:HDFGroup/hdf5.git
cd hdf5
git checkout hdf5_2.0.0
mkdir build
cd build
cmake -G “Unix Makefiles” -DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_TESTING:BOOL=ON -DHDF5_BUILD_TOOLS:BOOL=ON -DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=ON -DBUILD_SHARED_LIBS:BOOL=ON ..

and I could see that the build process picked up my system zlib rather than trying to download one:

– Looking for stdatomic.h
– Looking for stdatomic.h - found
– Found ZLIB: /usr/lib64/libz.so (found version “1.3.1”)
– H5_ZLIB_HEADER=zlib.h

I can also see that the resulting H5pubconf.h header correctly lists zlib support as being available (H5_HAVE_ZLIB_H and H5_HAVE_FILTER_DEFLATE both being defined).

Could you list the exact command you use to build so that we’re on the same page and also maybe attach the CMakeCache.txt file from your build directory so that I can take a look? I suspect there is something still being cached that is causing the build process to try and get an external zlib.

7

That’s exactly how I would expect ours to build, as well as where libz.so is installed. I am using the exact command that you had suggested earlier, the only difference being the quotes around Unix Makefiles.

cmake -G "Unix Makefiles" -DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_TESTING:BOOL=ON -DHDF5_BUILD_TOOLS:BOOL=ON -DHDF5_ENABLE_ZLIB_SUPPORT:BOOL=ON -DHDF5_ENABLE_SZIP_SUPPORT:BOOL=ON -DSZIP_USE_EXTERNAL=ON -DHDF5_ALLOW_EXTERNAL_SUPPORT:STRING=TGZ -DLIBAEC_TGZ_NAME:STRING=libaec-1.1.3.tar.gz -DLIBAEC_TGZ_ORIGPATH:STRING=%{_hdf5_build_loc}/libaec/ -DLIBAEC_USE_LOCALCONTENT:BOOL=OFF -DBUILD_SHARED_LIBS:BOOL=ON ../hdf5-2.0.0

Attached should be the CMakeCache.txt. This was from a build that was interrupted because it tried to reach git with no internet access. This was in a fresh container, which creates the build directory and started completely empty.

CMakeCache.txt (49.9 KB)

Thanks again for responding and working with me on this. I am rather confused as to why I am having these issues, as it seems things should be in order.

8

Thanks for the updates, I can see the issue now after looking at your CMakeCache.txt file. For some reason, the current CMake logic that we have always forces ZLIB_USE_EXTERNAL=ON whenever HDF5_ALLOW_EXTERNAL_SUPPORT is GIT or TGZ, which probably doesn’t make sense in general. By asking for external support for libaec, external support for zlib is also getting forced on when it shouldn’t be and preventing usage of a system-installed zlib, as you suspected. Unfortunately, until we can fix this it means that for HDF5 2.0.0 you need to do one of:

  • Use both zlib and libaec from external sources, presumably a tar.gz in your case
  • Backport the changes in HDF5 develop for using a system-installed libaec to HDF5 2.0.0 and use both zlib and libaec from the system-installed versions
  • Use system-installed zlib and omit support for libaec

The upcoming HDF5 2.1.0 release should make the second option simple, but we’re still working on that and don’t have a defined timeline yet other than “soon”.

9

Well, at least I finally have an answer. Thank you so much for helping me with this today. I’ve been trying everything in the INSTALL_CMake.txt file wondering if I was just reading it incorrectly. For now I will look into using the tar.gz, and will revisit the system zlib in a future release.

Thanks again!

1 Like
10

Thanks as well for bringing the problem to our attention. We’ll look into having this fixed for the 2.1.0 release and we hope to update our documentation soon to be more helpful in general when it comes to building HDF5.

11

Hi again @joseph.kelmer,

with the simple changes in Remove force-setting of ZLIB_USE_EXTERNAL / SZIP_USE_EXTERNAL by jhendersonHDF ¡ Pull Request #6222 ¡ HDFGroup/hdf5 ¡ GitHub it should now be possible to build with a system zlib and libaec from external sources (or vice versa). This should make it into the 2.1.0 release. Thanks!

1 Like