HDF5_DISABLE_VERSION_CHECK and disable_version_check

Could someone explain me the logic behind the version check ?

Or two sequential versions have compatible API or they don't have.

hdf5-1.8.17 and hdf5-1.8.16 both produces the same API version

usr/bin/cyghdf5-10.dll
usr/bin/cyghdf5_hl-10.dll

I don't understand the logic to state that they have the same API
but scare the users:

···

-------------------------------------------------------------------
Warning! ***HDF5 library version mismatched error***
The HDF5 header files used to compile this application do not match
the version used by the HDF5 library to which this application is linked.
Data corruption or segmentation faults may occur if the application continues.
This can happen when an application was compiled by one version of HDF5 but
linked with a different version of static or shared HDF5 library.
You should recompile the application or check your shared library related
settings such as 'LD_LIBRARY_PATH'.
'HDF5_DISABLE_VERSION_CHECK' environment variable is set to 1, application will
continue at your own risk.
--------------------------------------------------------------------

As example of puzzled users:
https://cygwin.com/ml/cygwin/2016-06/msg00267.html

I am not really a fan of rebuilding every program depending from HDF5
if the API is still compatible, it is against the reason of having a
API version.

Thanks in advance
Marco Atzeri

(cygwin package maintainer)

My understanding is that releases of HDF5 which differ in only the last (patch) digit are not *always* ABI (Application Binary Interface) compatible. They try to be but for practical reasons they are not always.

When the are not, compiling (e.g. using headers) with one version of the library but linking (e.g. using .a,.so,.dyllib) to another can fail for all sorts of hard-to-diagnose reasons.

The version check HDF5 has implemented is a sufficient but not always necessary check. It always prevent potentially bad things from happening. But, that is at the expense of not allowing things to proceed when they may be able to.

The HDF5_DISABLE_VERSION_CHECK env. variable setting is a way to work-around the not-always-necessary cases.

But, all that said, the issue here is that HDF5 developers are leaving it up to users of the library to decide if two versions are ABI compatible. Or, more correctly, they make no claim that two last-digit-differing versions will be ABI compatible. If two versions are, or at least are within the confines of the subset of calls made by a particular application, that is more by coincidence then by design. At least that is my impression.

OTOH, HDF5 develepers could build-in to the library knowledge of its ABI compatible version history so that when headers and libs from two different versions are mixed, it generates an error only when the two are indeed known not to be ABI compatible.

Mark

PS: In at least one case in the past (1.6.0 to 1.6.2 I think) two different versions were not even file format compatible if you were using checksuming feature.

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of Marco Atzeri <marco.atzeri@gmail.com<mailto:marco.atzeri@gmail.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, June 21, 2016 at 1:54 AM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] HDF5_DISABLE_VERSION_CHECK and disable_version_check

Could someone explain me the logic behind the version check ?

Or two sequential versions have compatible API or they don't have.

hdf5-1.8.17 and hdf5-1.8.16 both produces the same API version

usr/bin/cyghdf5-10.dll
usr/bin/cyghdf5_hl-10.dll

I don't understand the logic to state that they have the same API
but scare the users:

-------------------------------------------------------------------
Warning! ***HDF5 library version mismatched error***
The HDF5 header files used to compile this application do not match
the version used by the HDF5 library to which this application is linked.
Data corruption or segmentation faults may occur if the application
continues.
This can happen when an application was compiled by one version of HDF5 but
linked with a different version of static or shared HDF5 library.
You should recompile the application or check your shared library related
settings such as 'LD_LIBRARY_PATH'.
'HDF5_DISABLE_VERSION_CHECK' environment variable is set to 1,
application will
continue at your own risk.
--------------------------------------------------------------------

As example of puzzled users:
https://cygwin.com/ml/cygwin/2016-06/msg00267.html

I am not really a fan of rebuilding every program depending from HDF5
if the API is still compatible, it is against the reason of having a
API version.

Thanks in advance
Marco Atzeri

(cygwin package maintainer)

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

My understanding is that releases of HDF5 which differ in only the last
(patch) digit are not *always* ABI (Application Binary Interface)
compatible. They try to be but for practical reasons they are not always.

When they are not compatible, they should bump the ABI.
HDF5 it is already a champion as bumper in that field,
I am not aware of another library that bumped so many
time between minor version.

When the are not, compiling (e.g. using headers) with one version of the
library but linking (e.g. using .a,.so,.dyllib) to another can fail for
all sorts of hard-to-diagnose reasons.

The normal case on distribution is different:

I built the octave and netcdf package with hdf5-1.8.16;
later I package the hdf5-1.8.17 that is supposed to be ABI compatible,
and it produces a shared lib with the same name of hdf5-1.8.16 release.

Why I should rebuild octave and netcdf ?
It does not make sense.

Regards
Marco

···

On 21/06/2016 14:33, Miller, Mark C. wrote:

I think it makes sense for any number of possible reasons…

  * You occasionally want to support changes to the API in patch releases that break ABI compatability
  * Your release frequency is too low (or, equivalently, the energy barrier to make an official release is too high)
  * You don’t really have the resources to support wholly decoupled releases of just bug-fixes and new development
  * You want to control perceptions regarding how much different two versions are that differ in major and minor digits

But, you are right, it is not truly consistent with a large number of other projects version numbering practices.

There are probably subtle issues I’ve failed to understand or explain too.

Mark

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of Marco Atzeri <marco.atzeri@gmail.com<mailto:marco.atzeri@gmail.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, June 21, 2016 at 7:17 AM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] HDF5_DISABLE_VERSION_CHECK and disable_version_check

On 21/06/2016 14:33, Miller, Mark C. wrote:
My understanding is that releases of HDF5 which differ in only the last
(patch) digit are not *always* ABI (Application Binary Interface)
compatible. They try to be but for practical reasons they are not always.

When they are not compatible, they should bump the ABI.
HDF5 it is already a champion as bumper in that field,
I am not aware of another library that bumped so many
time between minor version.

When the are not, compiling (e.g. using headers) with one version of the
library but linking (e.g. using .a,.so,.dyllib) to another can fail for
all sorts of hard-to-diagnose reasons.

The normal case on distribution is different:

I built the octave and netcdf package with hdf5-1.8.16;
later I package the hdf5-1.8.17 that is supposed to be ABI compatible,
and it produces a shared lib with the same name of hdf5-1.8.16 release.

Why I should rebuild octave and netcdf ?
It does not make sense.

Regards
Marco

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Marco,

Thank you for your feedback! Well deserved critique!

You are absolutely correct that this is an annoying problem that has to be addressed. Mark provided excellent explanation, but it does not help you to maintain the packages built on top of HDF5.

We started addressing the issue for 1.8.17, but couldn’t finish the work due to very limited bandwidth, complexity of the issue and disagreement on approach to the solution. I will not attempt to explain it in this email. The problem deserves a well written document that we don’t have at this point (contributes to the delay in finishing the work ;-).

I just wanted to let everyone know that we are very well aware of the problem; it has a high priority and we will get back to it after all features planned for 1.10.0 are finally out this Fall.

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Jun 21, 2016, at 9:17 AM, Marco Atzeri <marco.atzeri@gmail.com<mailto:marco.atzeri@gmail.com>> wrote:

On 21/06/2016 14:33, Miller, Mark C. wrote:
My understanding is that releases of HDF5 which differ in only the last
(patch) digit are not *always* ABI (Application Binary Interface)
compatible. They try to be but for practical reasons they are not always.

When they are not compatible, they should bump the ABI.
HDF5 it is already a champion as bumper in that field,
I am not aware of another library that bumped so many
time between minor version.

When the are not, compiling (e.g. using headers) with one version of the
library but linking (e.g. using .a,.so,.dyllib) to another can fail for
all sorts of hard-to-diagnose reasons.

The normal case on distribution is different:

I built the octave and netcdf package with hdf5-1.8.16;
later I package the hdf5-1.8.17 that is supposed to be ABI compatible,
and it produces a shared lib with the same name of hdf5-1.8.16 release.

Why I should rebuild octave and netcdf ?
It does not make sense.

Regards
Marco

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Question:
if I build 1.8.17 again with disable_version_check=1 is fine or not ?

I was not planning to build any 1.10.x version short term.

Regards
Marco

···

On 22/06/2016 06:42, Elena Pourmal wrote:

Marco,

Thank you for your feedback! Well deserved critique!

You are absolutely correct that this is an annoying problem that has to
be addressed. Mark provided excellent explanation, but it does not help
you to maintain the packages built on top of HDF5.

We started addressing the issue for 1.8.17, but couldn�t finish the work
due to very limited bandwidth, complexity of the issue and disagreement
on approach to the solution. I will not attempt to explain it in this
email. The problem deserves a well written document that we don�t have
at this point (contributes to the delay in finishing the work ;-).

I just wanted to let everyone know that we are very well aware of the
problem; it has a high priority and we will get back to it after all
features planned for 1.10.0 are finally out this Fall.

Elena