Dear HDF5 community,
I plan to use hdf5 to save large amount of data saving disk space. I am able to write the HDF5 file (fortran) but first I noticed that despite using compression there was no improvement in file size compare to flat binary.
After inquiring in the documentation resources available in the HDF5 website and in the internet, I figured out that it might be that szip and zlib libraries were not found.
Then I compiled by myself, following the installation instructions, I compiled szip, zlib and hdf5
export CC=icc
export CFLAGS='-O3 -xHost -ip'
tar xf szip-2.1.1.tar
cd szip-2.1.1
./configure --prefix=/home/apps/hdf5/szip/v2.1.1
make
make check
make install
export CC=icc
export CFLAGS='-O3 -xHost -ip'
tar xf zlib-1.2.13.tar
cd zlib-1.2.13
./configure --prefix=/home/apps/hdf5/zlib/v1.2.13
make
make check
make install
export CC=icc
export FC=ifort
export CXX=icpc
tar -xf hdf5-1.14.1-2.tar
cd hdf5-1.14.1-2
./configure --prefix=/home/apps/hdf5/v1.14 --enable-fortran \
--enable-cxx --enable-hl --with-szlib=/home/apps/hdf5/szip/v2.1.1/ \
--with-zlib=/home/apps/hdf5/zlib/v1.2.13/ \
--enable-using-memchecker
make -j 8 && make -j 8 check
make -j 8 install
The three libraries are built successfully and all the tests passed.
But when I run my program and it create the h5 file, I still did not see any filter apply during compression: I check the file using h5dump -pH
And you can see that the output of h5fc -show, it is telling that the libraries should be found
ifort -I/home/apps/hdf5/v1.14/include -L/home/apps/hdf5/v1.14/lib /home/apps/hdf5/v1.14/lib/libhdf5hl_fortran.a /home/apps/hdf5/v1.14/lib/libhdf5_hl.a /home/apps/hdf5/v1.14/lib/libhdf5_fortran.a /home/apps/hdf5/v1.14/lib/libhdf5.a -L/home/apps/hdf5/zlib/v1.2.13/lib -lz -ldl -lm -Wl,-rpath -Wl,/home/apps/hdf5/v1.14/lib
Note: the h5fc output shows only zlib because at that this time I haven’t specify --with-szib option.
So the first issue is: that zlib compression seems not to be used at all when a write a hdf5 file following the example “Create a chunked and compressed dataset”
h5dump -pH shows FILTERS = NONE
The second issue is that I have not succeed to configure hdf5 with szip encoding compression and I read in this forum that I should build with both endode/decode filters. I have checked in the compilation of szip that it was correctly build with encoding. In the config.log file it shows #define HAVE_ENCODING 1
which I’m interpreting as yes.
I read elsewhere that this might be a bug of configure script that detect the encoding is missing when it is actually present.
So I am a bit worried about this situation and I decided to write here.
Please let me know if I am missing some important detail, otherwise why is this happening. As I said one of the main reason I plan to use HDF5 is for data compression which is important for the problem I am solving.
Finally, I want to give some details about the data I am saving.
It is just four 2D arrays having dimensions: (N,M), (N,M), (1,M), (1,M)
I haven’t tried with large values of M and N, as I would like to do, and for the test I have N = 7665 and M = 2015
Kind regards
Manuel