HDF5 Version 1.8.4 Requires Zlib and SZip ?

Hdf-forum archives
1

I tried the following:

./configure --without-zlib --without-szlib --disable-hl --disable-static --disable-cxx --disable-filters

and then ran "make test" and I get failures in the tsohm.c file due to lack of filters available. With the 1.6 release I used to be able to build it this way without any problems. Now it seems that I _must_ link against at least one of the zlib or szlib libraries? Is this correct?

This is holding up the CMake port at the moment. Just trying to figure out if this is a valid configuration.

Thanks

···

_________________________________________________________
Mike Jackson mike.jackson@bluequartz.net
BlueQuartz Software www.bluequartz.net
Principal Software Engineer Dayton, Ohio

2

Hi Mike and All,

You guys are amazing! It looks like things are moving quickly and we will have to deal with Cmake before Christmas :slight_smile:

In HDF5 zlib and szip filters are optional; users have to specify szip, but for zlib if is different - it is added when configure finds it on the system. Other filters (nbit, fletcher32, scaleoffset, and shuffle) are included by default, but can be disabled at the configuration time by using --enable-filters flag.

Here is an excerpt from the H5pubconf.h file when I configure on Linux 32-bit without specifying location of the szip library, (zlib is available on the system):

/* Define if support for deflate (zlib) filter is enabled */
#define H5_HAVE_FILTER_DEFLATE 1

/* Define if support for Fletcher32 checksum is enabled */
#define H5_HAVE_FILTER_FLETCHER32 1

/* Define if support for nbit filter is enabled */
#define H5_HAVE_FILTER_NBIT 1

/* Define if support for scaleoffset filter is enabled */
#define H5_HAVE_FILTER_SCALEOFFSET 1

/* Define if support for shuffle filter is enabled */
#define H5_HAVE_FILTER_SHUFFLE 1

/* Define if support for szip filter is enabled */
/* #undef H5_HAVE_FILTER_SZIP */

I am not sure what you need to do in Cmake; variables shown above have to be defined/undefined depending on what a user wants to use (note: in this example, zlib is available, therefore H5_HAVE_FILTER_DEFLATE is set to 1; if for some reason it was not found, then the variable would be not defined)

Source of each files

H5Zdeflate.c
H5Zfletcher32.c
H5Zshuffle.c
H5Zszip.c
H5Znbit.c
H5Zscaleoffset.c

contains the corresponding #ifdef H5_HAVE_FILTER_* statement.

If the corresponding variable is not set up, then source is not compiled in. I guess, Cmake has to set H5_HAVE_FILTER_* to the correct value.

Did I answer your question?

What you reported sounds like a bug to me. Our tests should be aware which filters are not available and act accordingly.

Thanks again for your efforts!

Elena

···

On Nov 18, 2009, at 6:08 PM, Michael Jackson wrote:

I tried the following:

./configure --without-zlib --without-szlib --disable-hl --disable-static --disable-cxx --disable-filters

and then ran "make test" and I get failures in the tsohm.c file due to lack of filters available. With the 1.6 release I used to be able to build it this way without any problems. Now it seems that I _must_ link against at least one of the zlib or szlib libraries? Is this correct?

This is holding up the CMake port at the moment. Just trying to figure out if this is a valid configuration.

Thanks

_________________________________________________________
Mike Jackson mike.jackson@bluequartz.net
BlueQuartz Software www.bluequartz.net
Principal Software Engineer Dayton, Ohio

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

3

CMake can handle getting the options from the user and looking for the various libraries and then adding in the necessary source files. Currently it seems that by just setting the "H5_HAVE_FILTER_*" is enough to either enable the filter or disable the filter as the library will compile but the tests fail because the number of filters registered is 0.

   The Current state of the CMake system for 1.8 is still rough but is starting to be workable. We will see what John B. does with it over night.

  I still have the szip code as a subproject of the HDF5 1.8 code base which seems like it probably is NOT correct due to licensing. Not sure. Someone will have to get a ruling on that. Someone will pretty quickly add in the correct CMake code to just look on the system for szip and zlib instead of having to build them. I think that may be already in for ZLib. SZip was pretty shaky when I committed tonight.

   If you want to follow along on the progress there is an RSS feed that you can subscribe to that gets updated everytime someone commits to the repo.

   If someone could look into the bug that would be great to figure out if I am simply doing something incorrect or not.

Lastly,
    How does The HDF Group feel about making some minor code changes to some of the source files during this process. I have had to make a few changes to have some preprocessor statements be more consistent.

  I also spent a bunch of time moving the necessary changes to H5detect.c over to take into account the 4 architectures on OS X. Moving that code into the mainline HDF5 code base would be well worth it.

Mike Jackson

···

On Nov 18, 2009, at 9:22 PM, Elena Pourmal wrote:

Hi Mike and All,

You guys are amazing! It looks like things are moving quickly and we will have to deal with Cmake before Christmas :slight_smile:

In HDF5 zlib and szip filters are optional; users have to specify szip, but for zlib if is different - it is added when configure finds it on the system. Other filters (nbit, fletcher32, scaleoffset, and shuffle) are included by default, but can be disabled at the configuration time by using --enable-filters flag.

Here is an excerpt from the H5pubconf.h file when I configure on Linux 32-bit without specifying location of the szip library, (zlib is available on the system):

/* Define if support for deflate (zlib) filter is enabled */
#define H5_HAVE_FILTER_DEFLATE 1

/* Define if support for Fletcher32 checksum is enabled */
#define H5_HAVE_FILTER_FLETCHER32 1

/* Define if support for nbit filter is enabled */
#define H5_HAVE_FILTER_NBIT 1

/* Define if support for scaleoffset filter is enabled */
#define H5_HAVE_FILTER_SCALEOFFSET 1

/* Define if support for shuffle filter is enabled */
#define H5_HAVE_FILTER_SHUFFLE 1

/* Define if support for szip filter is enabled */
/* #undef H5_HAVE_FILTER_SZIP */

I am not sure what you need to do in Cmake; variables shown above have to be defined/undefined depending on what a user wants to use (note: in this example, zlib is available, therefore H5_HAVE_FILTER_DEFLATE is set to 1; if for some reason it was not found, then the variable would be not defined)

Source of each files

H5Zdeflate.c
H5Zfletcher32.c
H5Zshuffle.c
H5Zszip.c
H5Znbit.c
H5Zscaleoffset.c

contains the corresponding #ifdef H5_HAVE_FILTER_* statement.

If the corresponding variable is not set up, then source is not compiled in. I guess, Cmake has to set H5_HAVE_FILTER_* to the correct value.

Did I answer your question?

What you reported sounds like a bug to me. Our tests should be aware which filters are not available and act accordingly.

Thanks again for your efforts!

Elena

On Nov 18, 2009, at 6:08 PM, Michael Jackson wrote:

I tried the following:

./configure --without-zlib --without-szlib --disable-hl --disable-static --disable-cxx --disable-filters

and then ran "make test" and I get failures in the tsohm.c file due to lack of filters available. With the 1.6 release I used to be able to build it this way without any problems. Now it seems that I _must_ link against at least one of the zlib or szlib libraries? Is this correct?

This is holding up the CMake port at the moment. Just trying to figure out if this is a valid configuration.

Thanks

_________________________________________________________
Mike Jackson mike.jackson@bluequartz.net
BlueQuartz Software www.bluequartz.net
Principal Software Engineer Dayton, Ohio

4

Mike,

CMake can handle getting the options from the user and looking for the various libraries and then adding in the necessary source files. Currently it seems that by just setting the "H5_HAVE_FILTER_*" is enough to either enable the filter or disable the filter as the library will compile but the tests fail because the number of filters registered is 0.

The Current state of the CMake system for 1.8 is still rough but is starting to be workable. We will see what John B. does with it over night.

I still have the szip code as a subproject of the HDF5 1.8 code base which seems like it probably is NOT correct due to licensing. Not sure. Someone will have to get a ruling on that. Someone will pretty quickly add in the correct CMake code to just look on the system for szip and zlib instead of having to build them. I think that may be already in for ZLib. SZip was pretty shaky when I committed tonight.

In the past, we had zlib as a part of the HDF4 distribution since zlib (and jpeg) is a required component of HDF4. After about 15 years of maintenance, we decided that we didn't want to maintain and sync zlib (and jpeg) code with the official distribution, therefore we removed it from the HDF4 source and have been using it as an external library since. I think we should follow the trend and treat both zlib and szip as external projects to avoid licensing and maintenance issues (and both libraries are OPTIONAL).

If you want to follow along on the progress there is an RSS feed that you can subscribe to that gets updated everytime someone commits to the repo.

If someone could look into the bug that would be great to figure out if I am simply doing something incorrect or not.

Lastly,
  How does The HDF Group feel about making some minor code changes to some of the source files during this process. I have had to make a few changes to have some preprocessor statements be more consistent.

I also spent a bunch of time moving the necessary changes to H5detect.c over to take into account the 4 architectures on OS X. Moving that code into the mainline HDF5 code base would be well worth it.

Please send us your changes for review.

Thank you!

Elena

···

On Nov 18, 2009, at 8:56 PM, Michael Jackson wrote:

Mike Jackson

On Nov 18, 2009, at 9:22 PM, Elena Pourmal wrote:

Hi Mike and All,

You guys are amazing! It looks like things are moving quickly and we will have to deal with Cmake before Christmas :slight_smile:

In HDF5 zlib and szip filters are optional; users have to specify szip, but for zlib if is different - it is added when configure finds it on the system. Other filters (nbit, fletcher32, scaleoffset, and shuffle) are included by default, but can be disabled at the configuration time by using --enable-filters flag.

Here is an excerpt from the H5pubconf.h file when I configure on Linux 32-bit without specifying location of the szip library, (zlib is available on the system):

/* Define if support for deflate (zlib) filter is enabled */
#define H5_HAVE_FILTER_DEFLATE 1

/* Define if support for Fletcher32 checksum is enabled */
#define H5_HAVE_FILTER_FLETCHER32 1

/* Define if support for nbit filter is enabled */
#define H5_HAVE_FILTER_NBIT 1

/* Define if support for scaleoffset filter is enabled */
#define H5_HAVE_FILTER_SCALEOFFSET 1

/* Define if support for shuffle filter is enabled */
#define H5_HAVE_FILTER_SHUFFLE 1

/* Define if support for szip filter is enabled */
/* #undef H5_HAVE_FILTER_SZIP */

I am not sure what you need to do in Cmake; variables shown above have to be defined/undefined depending on what a user wants to use (note: in this example, zlib is available, therefore H5_HAVE_FILTER_DEFLATE is set to 1; if for some reason it was not found, then the variable would be not defined)

Source of each files

H5Zdeflate.c
H5Zfletcher32.c
H5Zshuffle.c
H5Zszip.c
H5Znbit.c
H5Zscaleoffset.c

contains the corresponding #ifdef H5_HAVE_FILTER_* statement.

If the corresponding variable is not set up, then source is not compiled in. I guess, Cmake has to set H5_HAVE_FILTER_* to the correct value.

Did I answer your question?

What you reported sounds like a bug to me. Our tests should be aware which filters are not available and act accordingly.

Thanks again for your efforts!

Elena

On Nov 18, 2009, at 6:08 PM, Michael Jackson wrote:

I tried the following:

./configure --without-zlib --without-szlib --disable-hl --disable-static --disable-cxx --disable-filters

and then ran "make test" and I get failures in the tsohm.c file due to lack of filters available. With the 1.6 release I used to be able to build it this way without any problems. Now it seems that I _must_ link against at least one of the zlib or szlib libraries? Is this correct?

This is holding up the CMake port at the moment. Just trying to figure out if this is a valid configuration.

Thanks

_________________________________________________________
Mike Jackson mike.jackson@bluequartz.net
BlueQuartz Software www.bluequartz.net
Principal Software Engineer Dayton, Ohio

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

5

We will see what John B. does with it over
night.

NB. If the hdf people would prefer us to not post

Had to work on something else this morning, but the dynamic lib building was broken for win32, so I have made a couple of changes. Just FYI

This works fine for setting the export dll flags in the hdf5 dll
SET_PROPERTY(TARGET hdf5 APPEND PROPERTY COMPILE_DEFINITIONS _HDF5_HLDLL_EXPORTS_)

But you also need to add _HDF5USEDLL_ to every target that links against it - which includes those outside of the project to force dll import status, the easiest way is simply to skip the target properties and add the stuff below to the config.cmake.in The "BUILD_SHARED_LIBS" is replaced by H5_xxx in configure and "hdf5_EXPORTS" is set by cmake for the library itself, and for all other projects we get the use dll alternative if build shared is set. (I know Mike already know this stuff, but maybe the others are interested - incidentally, if cmakeification messages are cluttering the list up undesirably, then please say and we'll keep cmake related stuff to ourselves unless we need clarification of anything).

#cmakedefine BUILD_SHARED_LIBS
#if defined(H5_BUILD_SHARED_LIBS)
# if defined (hdf5_EXPORTS)
# define _HDF5DLL_
# else
# define _HDF5USEDLL_
# endif
#endif

Note also : Mike - I see that you changed the WINDOWS_MAX_BUF to H5_WINDOWS_MAX_BUF in the hdf5 source code - fine by me, but it might be a good idea to make sure that any changes we make to hdf5 code itself outside of cmake scripts are brought to the attention of the maintainers in case they lose diffs.

JB

6

Mike

Erratum ... I noticed that when turning testing on, when using dynamic libs, the same trouble appears with the testing lib, so I've added a couple more bits to the config to do testing at the same time.

I'm impressed. 50 targets all building with errors. You did a good job. Time to start a dashboard methinks. I've got pretty good test results, but some fails

Test project D:/cmakebuild/hdf-1.8.4
  1/ 34 Testing testhdf5 ***Failed
  2/ 34 Testing lheap Passed
  3/ 34 Testing ohdr Passed
  4/ 34 Testing stab Passed
  5/ 34 Testing gheap Passed
  6/ 34 Testing cache Passed
  7/ 34 Testing hyperslab Passed
  8/ 34 Testing istore Passed
  9/ 34 Testing bittests Passed
10/ 34 Testing dtypes ***Failed
11/ 34 Testing dsets Passed
12/ 34 Testing cmpd_dset Passed
13/ 34 Testing extend Passed
14/ 34 Testing external Passed
15/ 34 Testing links ***Failed
16/ 34 Testing unlink Passed
17/ 34 Testing big Passed
18/ 34 Testing mtime Passed
19/ 34 Testing fillval Passed
20/ 34 Testing mount Passed
21/ 34 Testing flush1 Passed
22/ 34 Testing flush2 Passed
23/ 34 Testing enum Passed
24/ 34 Testing set_extent Passed
25/ 34 Testing ttsafe Passed
26/ 34 Testing getname ***Failed
27/ 34 Testing ntypes ***Failed
28/ 34 Testing dangle Passed
29/ 34 Testing h5diffgentest ***Failed
30/ 34 Testing h5dumpgentest ***Failed
31/ 34 Testing h5jamgentest ***Failed
32/ 34 Testing h5importtest Passed
33/ 34 Testing h5repacktest ***Failed
34/ 34 Testing testh5repack_detect_szip Passed

74% tests passed, 9 tests failed out of 34

JB

7

All,
    I find all this discussion very enlightening!
As I have the responsibility for testing and maintaining the HDF 4/5/Java products on Windows, please, copy me in on any discussions. I will actively start using the cmake work you all have been doing right after Thanksgiving (I need to finish some other projects), and once I get familiar enough with cmake I can respond with the issues I'll need to address.

The one thing we do for testing is that we run all the Windows configurations as virtual machines under VMware. I will need to adjust our test program to control the cmake process. So please keep the messages coming!

Allen Byrne
byrn@hdfgroup.org

···

> We will see what John B. does with it over
> night.

NB. If the hdf people would prefer us to not post

(I know Mike already know this stuff, but maybe the others are interested - incidentally, if cmakeification messages are cluttering the list >up undesirably, then please say and we'll keep cmake related stuff to ourselves unless we need clarification of anything).

JB

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

8

Well, after an "interesting" week with HDF5 v1.8.4, CMake and Gitorious I now have about 3000 tests passing for OS X 10.5.8, 32 bit, Intel, Debug mode. The VFD test is not passing as it can not seem to find a file called "new_family_0004.h5" on the system to load.

   Jeff B. added some more options for the user to select (like having 1.6 API) and I added a bunch to test the conversion ability of int <-- > floats and all that.

You can take a look at the source code at the gitorious.com/hdf5 web site. I have a my.cdash.org account setup to submit testing results to. Just need to find the time to setup the CTest submission.

···

_________________________________________________________
Mike Jackson mike.jackson@bluequartz.net
BlueQuartz Software www.bluequartz.net
Principal Software Engineer Dayton, Ohio

On Nov 20, 2009, at 11:05 AM, Allen D Byrne wrote:

All,
   I find all this discussion very enlightening!
As I have the responsibility for testing and maintaining the HDF 4/5/Java products on Windows, please, copy me in on any discussions. I will actively start using the cmake work you all have been doing right after Thanksgiving (I need to finish some other projects), and once I get familiar enough with cmake I can respond with the issues I'll need to address.

The one thing we do for testing is that we run all the Windows configurations as virtual machines under VMware. I will need to adjust our test program to control the cmake process. So please keep the messages coming!

Allen Byrne
byrn@hdfgroup.org

We will see what John B. does with it over
night.

NB. If the hdf people would prefer us to not post

(I know Mike already know this stuff, but maybe the others are interested - incidentally, if cmakeification messages are cluttering the list >up undesirably, then please say and we'll keep cmake related stuff to ourselves unless we need clarification of anything).

JB

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org