HDFView altering files such that they cannot be read with IDL

Quincey_Koziol · December 2, 2009, 9:19pm

Hi Matt,

No one has explained a simple method to repair a file that has been
corrupted by HDFView2.5 with HDF5 1.8.2 or to even detect the
corruption: None of the h5 utilities detected it. It is pretty clear
to me that there is not much testing that happens for file
compatibilities across the **supported** API versions. I apologize
for sounding unappreciative fo the work done, and feel there is no
better alternative to HDF5, but is this really considered excellent?

      Maybe I missed this in your earlier messages, but was the file actually corrupted by
HDFView? Is it no longer readable? If so, could you provide access to a copy of it, so we
can work with you to address the issue.

Yes. HDFView2.5 corrupts HDF5 data files (whether created with h5py
and v1.8.* or IDL7.0) in such a way that they can not be read in IDL
at all.
  In IDL7.0, h5f_open() on a file that has been opened by HDFViewer fails.
  In IDL 6.3, h5f_open() on such a file crashes IDL.
  In IDL7.0, "x = h5_browser()" on such a file crashes IDL.

The file is altered in "the header" (sorry, I am new to hdf5 so do not
know the layout of the file), between bytes 160 and 192. More details
and small files prior to and after reading with HDFView2.5 to
demonstrate the problem are at
  http://cars9.uchicago.edu/pybeamline/DataFormats/H5UsageNotes/HDF5AndIDL

Ah, yes, the problem is that the bug in the HDF5 library [incorrectly] re-wrote the superblock for the HDF5 file (when opened with HDFView, or any other application that opened the file for read-write access) with a later version of the superblock than was previously there. So, the file was not corrupted (although I can certainly see that it appears so to a user), but was inadvertently upgraded to a later version of the file format, that IDL can't currently read. Any any case, it's fixed in the latest release of the HDF5 library (1.8.4).

The version information is contained in the files, that's not a problem.

Where? I thought that yesterday you said that not being able to
detect version numbers of a file was a feature, as multiple
applications may write to a single file. Perhaps I misunderstood you.
I don't see it, and none of the utilities report file versions or
object versions....

Each data structure (and often parts of data structures) in the HDF5 has its own version number. Each version number is updated independently of changes to other data structures. So, it's difficult to say that a particular HDF5 file conforms to a particular version of the HDF5 library - parts of it may be readable by very early library versions and parts of it may be readable by only the latest library version. This aligns with what I mentioned earlier about the library choosing the earliest possible version to write out - the granularity is very fine.

What I think you want is a utility that will check the format of each file to verify the version
of the objects within it, which I mentioned is on our development path. Is there something
else you'd like to see?

Rather than a separate utility, I would prefer to see that as each
object is opened for reading (by h5*_open), that a version number or
unique id tagging the version of that object would be read. If the
object ID is not recognized, it would be detectable as a "future
feature" that the library may not be able to use, so that the h5*_open
could fail gracefully. Perhaps this exists already? If different
APIs and formats are expected to be interchangeable, it seems like
you'd need some sort of check like this, no?

Yes, each API routine does this already and will return with an error if the library encounters a format version that it doesn't understand. Was there a particular instance which didn't do this?

Quincey

···

On Dec 2, 2009, at 12:48 PM, Matt Newville wrote:

andrew.collette · December 2, 2009, 10:28pm

Hi,

Just to provide another perspective... several of my users have also
complained about being unable to open (with IDL) files created by
various sub-versions of HDF5 1.8. I understand ITT is aware of these
problems and is trying to fix them. However, it remains that (1)
widely deployed third party software like IDL currently chokes on
files which use new features, and (2) there doesn't seem to be a
meaningful way to restrict the use of "new-format" data structures
apart from "compile against 1.6". This is actually kind of a problem
for me because by default I only distribute h5py with HDF5 1.8 on
Windows, and these files are not always readable from IDL.

- The H5Pset_libver_bounds() routine will be able to constrain the objects created/modified to conform to versions that can be read by different versions of the library.

Does this mean that if I call H5Pset_libver_bounds(fapl, lowversion,
highversion), HDF5 will *never* create a structure in the file with a
version greater than highversion? The documentation at the moment
suggests it will try to make a "best effort" to avoid such structures,
but if I accidentally use a feature that requires a new format, I run
the risk of it being rejected by a third-party application. In a
sense this isn't really THG's fault; application authors should be
prepared for errors related to versioning; I don't know why IDL has a
problem like this.

It would be nice if I could say when I create or open a file "use no
features introduced after version 1.6.7" (or even "in the 1.8
series"), and if I try to use something that requires a newer format,
have that API call simply fail. From my perspective (and that of a
number of my users), interoperability is the number one requirement,
even ahead of performance improvements and new features.

Andrew

Daniel_Kahn1 · December 2, 2009, 11:01pm

Quincey Koziol wrote:

  Yes. HDFView2.5 corrupts HDF5 data files (whether created with h5py
and v1.8.* or IDL7.0) in such a way that they can not be read in IDL
at all.
In IDL7.0, h5f_open() on a file that has been opened by HDFViewer fails.
In IDL 6.3, h5f_open() on such a file crashes IDL.
In IDL7.0, "x = h5_browser()" on such a file crashes IDL.
The file is altered in "the header" (sorry, I am new to hdf5 so do not
know the layout of the file), between bytes 160 and 192. More details
and small files prior to and after reading with HDFView2.5 to
demonstrate the problem are at

Ah, yes, the problem is that the bug in the HDF5 library [incorrectly] re-wrote the superblock for the HDF5 file (when opened with HDFView, or any other application that opened the file for read-write access) with a later version of the superblock than was previously there. So, the file was not corrupted (although I can certainly see that it appears so to a user), but was inadvertently upgraded to a later version of the file format, that IDL can't currently read. Any any case, it's fixed in the latest release of the HDF5 library (1.8.4).

The conclusion “IDL Currently can’t read” isn’t quite justified, since
Matt has presented data for only IDL6.3 and IDL7.0, but ITTVIS is up to
at least IDL7.1, which would be the “current” version of IDL.

We encountered a similar (perhaps the same) difficulty, whereby files
produced by early 1.8.x libraries were not readable by some, but not
all, 1.6.X versions of the library. Note that the whole file was
unreadable, not just selected objects. This was due to a bug (however,
not the bug Quincey refers to) and THG staff explained to me the how
the problem manifested itself. According to my notes, the break point
was that in 1.6.X the library did not exhibit the bug for X>=5. I
think upgrading IDL to 7.0.6 solved the problem for us, but I no longer
have access to 7.0.6, so I can’t check which DLM 7.0.6 used. IDL7.1
uses HDF5 1.6.7, and since 7>=5, it works. The HDF5 library in
IDL7.0 manifests the bug.

As I recall the bug which caused the problem was detected in early
version of 1.8.x and fixed in later ones, adding a further twist to
version number calculus, i.e one could solve the problem by upgrading
"reading" software to use HDF5 1.6.(x>=5) or “writing” software to
use HDF5 1.8.(x>=3). The situation was a bit messy to explain–at
the time I wrote that it was analogous to blood type incompatibilities.

The simple message to IDL users in our case was “use IDL version 7.1
or greater”. Indeed, I have tested Matt’s a.h5 file on IDL7.0 (fails)
and IDL7.1 (works). Also, for h5dump:

HDF5 1.6.3: fails;

HDF5 1.6.7: works

HDF5 1.8.1: works

(My recollection of the details are hazy, but I think there was a
change during the development of the 1.6 branch where a piece of
metadata was deemed unnecessary–it was still written out, but no
longer read. 1.8 branch developers stopped writing out this apparently
"unnecessary" bit of metadata, which caused older 1.6 branch libraries
to be fail. When the problem came to light they fixed it.)

Cheers,

–dan

···

http://cars9.uchicago.edu/pybeamline/DataFormats/H5UsageNotes/HDF5AndIDL

-- Daniel Kahn
Science Systems and Applications Inc.
301-867-2162

newville · December 3, 2009, 3:59am

The file is altered in "the header" (sorry, I am new to hdf5 so do not
know the layout of the file), between bytes 160 and 192. More details
and small files prior to and after reading with HDFView2.5 to
demonstrate the problem are at
http://cars9.uchicago.edu/pybeamline/DataFormats/H5UsageNotes/HDF5AndIDL

Ah, yes, the problem is that the bug in the HDF5 library [incorrectly] re-wrote the superblock for the
HDF5 file (when opened with HDFView, or any other application that opened the file for read-write
access) with a later version of the superblock than was previously there. So, the file was not
corrupted (although I can certainly see that it appears so to a user), but was inadvertently upgraded
to a later version of the file format, that IDL can't currently read. Any any case, it's fixed in the
latest release of the HDF5 library (1.8.4).

I hope you'll forgive me, but the word to describe what happened to
the file is "corrupted". Files which could be read by a working,
modern application can longer be read by that application after being
opened by a current version of HDFView. There is no warning from
HDFView and no sensible "unsupported version" message from any
application. Checking the file with HDF-supplied utilities showed no
differences, but a checksum on the file shows a change.

In any event, "fixed in the latest release of the library" does not
mean the problem is fixed in the current, supported version of
HDFView. (I haven't tested the beta version of HDFView: as with IDL
7.1, this is not the point). The problem with the underlying library
was apparently noted in August, and yet someone downloading current
HDF tools today would still see the problem.

No fix for the corrupted files has been suggested. Should one edit
the binary headers by hand?

Each data structure (and often parts of data structures) in the HDF5
has its own version number. Each version number is updated
independently of changes to other data structures. So, it's
difficult to say that a particular HDF5 file conforms to a
particular version of the HDF5 library - parts of it may be readable
by very early library versions and parts of it may be readable by
only the latest library version. This aligns with what I mentioned
earlier about the library choosing the earliest possible version to
write out - the granularity is very fine.

OK, that's somewhat reassuring. How do I get this information?
That is, how would one be able to distinguish a file being "corrupted"
and "inadvertently upgraded"? I can't even tell how to determine
which version of the library an application is using except to read
the Release notes for the application.

Yes, each API routine does this already and will return with an
error if the library encounters a format version that it doesn't
understand. Was there a particular instance which didn't do this?

I don't know. I don't see how to get this information. I was trying
to figure out if the data objects had changed. It appears that the
superblock had changed, not the data itself. Does the superblock have
a version? If so, how do I determine what it is?

If someone were using v1.8.4 of the API to write an application that
reads data from HDF5 files and wanted to guard against problems that
may arise trying to read features that might be added or changed in
v1.9 or v1.10 or later, what are your recommendations?

--Matt Newville <newville at cars.uchicago.edu>

···

On Wed, Dec 2, 2009 at 3:19 PM, Quincey Koziol <koziol@hdfgroup.org> wrote:

Quincey_Koziol · December 3, 2009, 3:34pm

Hi Andrew,

···

On Dec 2, 2009, at 4:28 PM, Andrew Collette wrote:

Hi,

Just to provide another perspective... several of my users have also
complained about being unable to open (with IDL) files created by
various sub-versions of HDF5 1.8. I understand ITT is aware of these
problems and is trying to fix them. However, it remains that (1)
widely deployed third party software like IDL currently chokes on
files which use new features, and (2) there doesn't seem to be a
meaningful way to restrict the use of "new-format" data structures
apart from "compile against 1.6". This is actually kind of a problem
for me because by default I only distribute h5py with HDF5 1.8 on
Windows, and these files are not always readable from IDL.

- The H5Pset_libver_bounds() routine will be able to constrain the objects created/modified to conform to versions that can be read by different versions of the library.

Does this mean that if I call H5Pset_libver_bounds(fapl, lowversion,
highversion), HDF5 will *never* create a structure in the file with a
version greater than highversion? The documentation at the moment
suggests it will try to make a "best effort" to avoid such structures,
but if I accidentally use a feature that requires a new format, I run
the risk of it being rejected by a third-party application. In a
sense this isn't really THG's fault; application authors should be
prepared for errors related to versioning; I don't know why IDL has a
problem like this.

It would be nice if I could say when I create or open a file "use no
features introduced after version 1.6.7" (or even "in the 1.8
series"), and if I try to use something that requires a newer format,
have that API call simply fail. From my perspective (and that of a
number of my users), interoperability is the number one requirement,
even ahead of performance improvements and new features.

Hmm, if H5Pset_libver_bounds() implies it's just a "best effort", then we should update the RM info. You are correct about the eventual behavior, we just didn't have time to finish things up before the 1.8 release needed to be finalized. We're planning on implementing the full functionality for the 1.10.0 release.

Quincey

newville · December 3, 2009, 3:38am

The conclusion "IDL Currently can't read" isn't quite justified, since Matt
has presented data for only IDL6.3 and IDL7.0, but ITTVIS is up to at least
IDL7.1, which would be the "current" version of IDL.

How would you suggest I control the version of IDL people use to read
HDF5 datasets?

--Matt

epourmal · December 3, 2009, 4:42am

Matt,

The file is altered in "the header" (sorry, I am new to hdf5 so do not
know the layout of the file), between bytes 160 and 192. More details
and small files prior to and after reading with HDFView2.5 to
demonstrate the problem are at
http://cars9.uchicago.edu/pybeamline/DataFormats/H5UsageNotes/HDF5AndIDL

Ah, yes, the problem is that the bug in the HDF5 library [incorrectly] re-wrote the superblock for the
HDF5 file (when opened with HDFView, or any other application that opened the file for read-write
access) with a later version of the superblock than was previously there. So, the file was not
corrupted (although I can certainly see that it appears so to a user), but was inadvertently upgraded
to a later version of the file format, that IDL can't currently read. Any any case, it's fixed in the
latest release of the HDF5 library (1.8.4).

I hope you'll forgive me, but the word to describe what happened to
the file is "corrupted". Files which could be read by a working,
modern application can longer be read by that application after being
opened by a current version of HDFView. There is no warning from
HDFView and no sensible "unsupported version" message from any
application. Checking the file with HDF-supplied utilities showed no
differences, but a checksum on the file shows a change.

In any event, "fixed in the latest release of the library" does not
mean the problem is fixed in the current, supported version of
HDFView. (I haven't tested the beta version of HDFView: as with IDL
7.1, this is not the point). The problem with the underlying library
was apparently noted in August, and yet someone downloading current
HDF tools today would still see the problem.

You have a very good point and I agree that we (The HDF Group) could be more proactive in raising awareness about the problem. Will try to do a better job next time.
We started working on the fix as soon as we learned about the issue; it took us several months to finish the work - it was not a trivial change to the library. New HDFView release is coming next.

We do as much as we can to test forward/backward compatibility considering our very limited resources, and continuos balance between library innovation and stability, and we learn as we go. We can definitely improve our testing procedures based on the new knowledge we acquired working with this community. It would be great if you (and anyone who is concerned with forward/backward compatibility) can provide us with the backward/forward compatibility test cases (and data files) that reflect your usage of HDF5. We will be glad to incorporate them in our daily regression testing routine.

No fix for the corrupted files has been suggested. Should one edit
the binary headers by hand?

Each data structure (and often parts of data structures) in the HDF5
has its own version number. Each version number is updated
independently of changes to other data structures. So, it's
difficult to say that a particular HDF5 file conforms to a
particular version of the HDF5 library - parts of it may be readable
by very early library versions and parts of it may be readable by
only the latest library version. This aligns with what I mentioned
earlier about the library choosing the earliest possible version to
write out - the granularity is very fine.

OK, that's somewhat reassuring. How do I get this information?

As Quincey mentioned, we are planning to improve in this area by creating new tool and adding new APIs that control object versioning.

That is, how would one be able to distinguish a file being "corrupted"
and "inadvertently upgraded"? I can't even tell how to determine
which version of the library an application is using except to read
the Release notes for the application.

I would suggest that you ask application people (IDL, etc.) to add this feature. HDF5 1.8.4 has it now by default - library information is embedded in the executables (use strings command to see the library version and how it was built).

Yes, each API routine does this already and will return with an
error if the library encounters a format version that it doesn't
understand. Was there a particular instance which didn't do this?

I don't know. I don't see how to get this information. I was trying
to figure out if the data objects had changed. It appears that the
superblock had changed, not the data itself. Does the superblock have
a version? If so, how do I determine what it is?

If someone were using v1.8.4 of the API to write an application that
reads data from HDF5 files and wanted to guard against problems that
may arise trying to read features that might be added or changed in
v1.9 or v1.10 or later, what are your recommendations?

If file created by 1.8.4 is modified with the new features introduced later, there is nothing we can do except to make sure that 1.8.4 handles this case gracefully.

Elena

···

On Dec 2, 2009, at 9:59 PM, Matt Newville wrote:

On Wed, Dec 2, 2009 at 3:19 PM, Quincey Koziol <koziol@hdfgroup.org> wrote:
--Matt Newville <newville at cars.uchicago.edu>

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Quincey_Koziol · December 3, 2009, 3:36pm

Yes, this was the exact problem with the file format, and it was compounded by the bug which accidentally re-wrote the file's superblock when the file was opened read-write but no changes were made. As I've said, both of these issues are corrected now.

Quincey

···

On Dec 2, 2009, at 5:01 PM, Daniel Kahn wrote:

Quincey Koziol wrote:

Yes. HDFView2.5 corrupts HDF5 data files (whether created with h5py
and v1.8.* or IDL7.0) in such a way that they can not be read in IDL
at all.
  In IDL7.0, h5f_open() on a file that has been opened by HDFViewer fails.
  In IDL 6.3, h5f_open() on such a file crashes IDL.
  In IDL7.0, "x = h5_browser()" on such a file crashes IDL.

The file is altered in "the header" (sorry, I am new to hdf5 so do not
know the layout of the file), between bytes 160 and 192. More details
and small files prior to and after reading with HDFView2.5 to
demonstrate the problem are at
  http://cars9.uchicago.edu/pybeamline/DataFormats/H5UsageNotes/HDF5AndIDL

  Ah, yes, the problem is that the bug in the HDF5 library [incorrectly] re-wrote the superblock for the HDF5 file (when opened with HDFView, or any other application that opened the file for read-write access) with a later version of the superblock than was previously there. So, the file was not corrupted (although I can certainly see that it appears so to a user), but was inadvertently upgraded to a later version of the file format, that IDL can't currently read. Any any case, it's fixed in the latest release of the HDF5 library (1.8.4).

The conclusion "IDL Currently can't read" isn't quite justified, since Matt has presented data for only IDL6.3 and IDL7.0, but ITTVIS is up to at least IDL7.1, which would be the "current" version of IDL.

We encountered a similar (perhaps the same) difficulty, whereby files produced by early 1.8.x libraries were not readable by some, but not all, 1.6.X versions of the library. Note that the whole file was unreadable, not just selected objects. This was due to a bug (however, not the bug Quincey refers to) and THG staff explained to me the how the problem manifested itself. According to my notes, the break point was that in 1.6.X the library did not exhibit the bug for X>=5. I think upgrading IDL to 7.0.6 solved the problem for us, but I no longer have access to 7.0.6, so I can't check which DLM 7.0.6 used. IDL7.1 uses HDF5 1.6.7, and since 7>=5, it works. The HDF5 library in IDL7.0 manifests the bug.

As I recall the bug which caused the problem was detected in early version of 1.8.x and fixed in later ones, adding a further twist to version number calculus, i.e one could solve the problem by upgrading "reading" software to use HDF5 1.6.(x>=5) or "writing" software to use HDF5 1.8.(x>=3). The situation was a bit messy to explain--at the time I wrote that it was analogous to blood type incompatibilities.

The simple message to IDL users in our case was "use IDL version 7.1 or greater". Indeed, I have tested Matt's a.h5 file on IDL7.0 (fails) and IDL7.1 (works). Also, for h5dump:

HDF5 1.6.3: fails;
HDF5 1.6.7: works
HDF5 1.8.1: works

(My recollection of the details are hazy, but I think there was a change during the development of the 1.6 branch where a piece of metadata was deemed unnecessary--it was still written out, but no longer read. 1.8 branch developers stopped writing out this apparently "unnecessary" bit of metadata, which caused older 1.6 branch libraries to be fail. When the problem came to light they fixed it.)

Quincey_Koziol · December 3, 2009, 3:45pm

Hi Matt,

The file is altered in "the header" (sorry, I am new to hdf5 so do not
know the layout of the file), between bytes 160 and 192. More details
and small files prior to and after reading with HDFView2.5 to
demonstrate the problem are at
http://cars9.uchicago.edu/pybeamline/DataFormats/H5UsageNotes/HDF5AndIDL

Ah, yes, the problem is that the bug in the HDF5 library [incorrectly] re-wrote the superblock for the
HDF5 file (when opened with HDFView, or any other application that opened the file for read-write
access) with a later version of the superblock than was previously there. So, the file was not
corrupted (although I can certainly see that it appears so to a user), but was inadvertently upgraded
to a later version of the file format, that IDL can't currently read. Any any case, it's fixed in the
latest release of the HDF5 library (1.8.4).

I hope you'll forgive me, but the word to describe what happened to
the file is "corrupted". Files which could be read by a working,
modern application can longer be read by that application after being
opened by a current version of HDFView. There is no warning from
HDFView and no sensible "unsupported version" message from any
application. Checking the file with HDF-supplied utilities showed no
differences, but a checksum on the file shows a change.

In any event, "fixed in the latest release of the library" does not
mean the problem is fixed in the current, supported version of
HDFView. (I haven't tested the beta version of HDFView: as with IDL
7.1, this is not the point). The problem with the underlying library
was apparently noted in August, and yet someone downloading current
HDF tools today would still see the problem.

No fix for the corrupted files has been suggested. Should one edit
the binary headers by hand?

I'm sorry I hadn't mentioned it earlier, but running h5repack on the file with the latest 1.8.x release will correct the issue in the output file it creates.

Each data structure (and often parts of data structures) in the HDF5
has its own version number. Each version number is updated
independently of changes to other data structures. So, it's
difficult to say that a particular HDF5 file conforms to a
particular version of the HDF5 library - parts of it may be readable
by very early library versions and parts of it may be readable by
only the latest library version. This aligns with what I mentioned
earlier about the library choosing the earliest possible version to
write out - the granularity is very fine.

OK, that's somewhat reassuring. How do I get this information?

Well, it's not normally exported at the application level, but the 'h5debug' tool can walk the file format information, if you are interested in the gory details.

That is, how would one be able to distinguish a file being "corrupted"
and "inadvertently upgraded"?

You can use the 'h5check' tool (ftp://ftp.hdfgroup.org/HDF5/special_tools/h5check/) to check if a file has been corrupted.

I can't even tell how to determine
which version of the library an application is using except to read
the Release notes for the application.

Yes, each API routine does this already and will return with an
error if the library encounters a format version that it doesn't
understand. Was there a particular instance which didn't do this?

I don't know. I don't see how to get this information. I was trying
to figure out if the data objects had changed. It appears that the
superblock had changed, not the data itself. Does the superblock have
a version? If so, how do I determine what it is?

Yes, the superblock has a version, and 'h5debug <filename>' should print it out for you.

If someone were using v1.8.4 of the API to write an application that
reads data from HDF5 files and wanted to guard against problems that
may arise trying to read features that might be added or changed in
v1.9 or v1.10 or later, what are your recommendations?

As Andrew mentioned in his last message, the H5Pset_libver_bounds() routine is planned to address this issue. A more cumbersome way to verify that an application is producing files that can be read by earlier versions of the HDF5 library is to compile the application against the earlier version of the library. If the application compiles successfully against earlier versions of the library, it will definitely produce files readable by that version of the library, even when compiled against later versions of the library. (modulo bugs of course).

Quincey

···

On Dec 2, 2009, at 9:59 PM, Matt Newville wrote:

On Wed, Dec 2, 2009 at 3:19 PM, Quincey Koziol <koziol@hdfgroup.org> wrote:

Daniel_Kahn1 · December 3, 2009, 3:12pm

Matt Newville wrote:

The conclusion "IDL Currently can't read" isn't quite justified, since Matt
has presented data for only IDL6.3 and IDL7.0, but ITTVIS is up to at least
IDL7.1, which would be the "current" version of IDL.

How would you suggest I control the version of IDL people use to read
HDF5 datasets?

Controlling people’s behavior is always a dicey proposition.

In this case you have two things in your favor:

Users will make some effort to read the data because they really
want it.
If users fail to comply the result isn’t likely to be a disaster.
No corrupt data is returned.

A number of NASA data producers also publish a document corresponding
to their data which describes the limitations and errors in their
data. On my previous project these were in documents named “README”,
presumably named after the habit of software producers to include a
file with this name which contained similar information about the
software.

I suggest that whatever mechanism you intended to use to communicate
this type of information to your users also includes any warnings or
caveats you are aware of regarding software versions, etc.

Cheers,

–dan

···

-- Daniel Kahn
Science Systems and Applications Inc.
301-867-2162

newville · December 3, 2009, 7:07pm

Hi Quincey,

No fix for the corrupted files has been suggested. Should one edit
the binary headers by hand?

I'm sorry I hadn't mentioned it earlier, but running h5repack on the file with
the latest 1.8.x release will correct the issue in the output file it creates.

Thanks -- that worked to fix the corrupted files.

Each data structure (and often parts of data structures) in the HDF5
has its own version number. Each version number is updated
independently of changes to other data structures. So, it's
difficult to say that a particular HDF5 file conforms to a
particular version of the HDF5 library - parts of it may be readable
by very early library versions and parts of it may be readable by
only the latest library version. This aligns with what I mentioned
earlier about the library choosing the earliest possible version to
write out - the granularity is very fine.

OK, that's somewhat reassuring. How do I get this information?

Well, it's not normally exported at the application level, but the
'h5debug' tool can walk the file format information, if you are interested
in the gory details.

The version information appears to be very new (to 1.8.4?). Is that
correct? As far as I can tell (from test files written with 1.8.4
and the docs), all the version numbers are currently 0. Is that also
correct? How would an application developer check these version
numbers when opening objects?

I'm a little confused about the H5Pset_libver_bounds() as it is
optional and would happen when writing a file, and so would probably
not be able to set an upper bound that was meaningful. Wouldn't it
be simpler and more robust for every object always have a version
number and let the reading library use this to decide if it recognizes
that version of that object?

--Matt Newville <newville at cars.uchicago.edu>

Quincey_Koziol · December 3, 2009, 8:33pm

Hi Matt,

Each data structure (and often parts of data structures) in the HDF5
has its own version number. Each version number is updated
independently of changes to other data structures. So, it's
difficult to say that a particular HDF5 file conforms to a
particular version of the HDF5 library - parts of it may be readable
by very early library versions and parts of it may be readable by
only the latest library version. This aligns with what I mentioned
earlier about the library choosing the earliest possible version to
write out - the granularity is very fine.

OK, that's somewhat reassuring. How do I get this information?

Well, it's not normally exported at the application level, but the
'h5debug' tool can walk the file format information, if you are interested
in the gory details.

The version information appears to be very new (to 1.8.4?). Is that
correct?

No, the version information has been there from the beginning.

As far as I can tell (from test files written with 1.8.4
and the docs), all the version numbers are currently 0. Is that also
correct?

Given that the files are probably being created with the "earliest version possible", it doesn't surprise me that they are using version 0 of things. If you use the H5Pset_libver_bounds(fapl, <latest>, <latest>) property setting, you should generate files with later version information.

How would an application developer check these version
numbers when opening objects?

Generally, they are not available at the application level.

I'm a little confused about the H5Pset_libver_bounds() as it is
optional and would happen when writing a file, and so would probably
not be able to set an upper bound that was meaningful. Wouldn't it
be simpler and more robust for every object always have a version
number and let the reading library use this to decide if it recognizes
that version of that object?

Yes, this is what actually occurs. H5Pset_libver_bounds() is designed to give the application developer some control over targeting the library versions that the file is planned to be read with.

Quincey

···

On Dec 3, 2009, at 1:07 PM, Matt Newville wrote:

newville · December 3, 2009, 10:16pm

OK, now I'm really confused. HDF5 objects do have version numbers,
and have always had version numbers, but they haven't been detectable
by an application until recently? Is that correct? I don't see
mention of an error that gets thrown due to a version mis-match.
Perhaps I'm missing this? I'm trying to understand how version
conflicts should be avoided. How do you recommend application
developers detect and avoid conflicts do to changes in the format?

All the versions numbers for files created by 1.8.4 (and h5py) are
reported to be 0, and the documentation doesn't seem to describe other
versions. Perhaps I'm missing this? The docs do mention
compatibility issues between v1.8 and v1.6 format files, suggesting
that there are other versions of objects. Are there versions of
objects other than 0? Where are they described?

Perhaps I am seeing version 0 for objects because I used h5py to
create the files, which actually uses the v1.6 API for compatibility
with older APIs, and specifically to suppress such conflicts. If that
is the case, all I can say is Thank You Very Very Very Much Andrew
Collette for saving us from non-detectable version conflicts!!

Thanks,

--Matt Newville <newville at cars.uchicago.edu>

Quincey_Koziol · December 4, 2009, 2:40am

Hi Matt,

OK, now I'm really confused. HDF5 objects do have version numbers,
and have always had version numbers, but they haven't been detectable
by an application until recently? Is that correct?

They've always had version numbers and they have never been exposed to the application directly. (i.e. the application can't say "what's the format version of this dataset?", see below for further details) Applications are supposed to check for error return values when calling HDF5 API routines.

I don't see
mention of an error that gets thrown due to a version mis-match.
Perhaps I'm missing this? I'm trying to understand how version
conflicts should be avoided. How do you recommend application
developers detect and avoid conflicts do to changes in the format?

Application developers should be prepared for any API call to fail, and handle that situation correctly. At a "low level" [in the application], failing to open an object due to changes to the file format should be handled like any other error. At a "high level" [planning out how data for a project will be stored in HDF5], the application developers or system architects need to create policies for which version of the HDF5 library will be used and how backward & forward format dependencies will be handled. The HDF5 library provides the H5Pset_libver_bounds() routine as a help in this area, but if you have some suggestions for improvements, we are always trying to improve the system.

All the versions numbers for files created by 1.8.4 (and h5py) are
reported to be 0, and the documentation doesn't seem to describe other
versions. Perhaps I'm missing this? The docs do mention
compatibility issues between v1.8 and v1.6 format files, suggesting
that there are other versions of objects. Are there versions of
objects other than 0? Where are they described?

The HDF5 file format, with all version information, is documented here:

http://www.hdfgroup.org/HDF5/doc/H5.format.html

I am getting the feeling that you want a particular version number on a dataset that says "this is version 3 of the dataset format". If that's the case, you won't find it, because we version the pieces of the format individually and at a finer granularity than "a dataset" or "a group". We version individual pieces of metadata, like B-tree nodes, object header messages, etc. Then, these pieces of metadata are composed together to create a higher-level object that the library exposes as "a dataset" or "a group" to an application. For example one dataset in a file might have a version 0 object header, with a version 2 datatype object header message and a version 1 chunk index, while another dataset in the file might have a version 1 object header, with version 1 datatype object header message and a version 0 contiguous dataset storage object header message, all depending on how an application created those datasets.

Perhaps I am seeing version 0 for objects because I used h5py to
create the files, which actually uses the v1.6 API for compatibility
with older APIs, and specifically to suppress such conflicts. If that
is the case, all I can say is Thank You Very Very Very Much Andrew
Collette for saving us from non-detectable version conflicts!!

Yes, that's probably the case why you aren't seeing version conflicts currently. h5py is not immune to this situation however, it's just that you are linking with the latest [1.8.x] library currently. I believe that Andrew is planning on creating wrappers for the new API routines in the 1.8.x release, which will complicate your life again. Also, if your version of h5py still linked against 1.8.x after the 1.10.x releases were out and you received a file with format changes from that release, you will have similar problems to the IDL situation.

Quincey

···

On Dec 3, 2009, at 4:16 PM, Matt Newville wrote:

andrew.collette · December 4, 2009, 4:00am

   Yes, that&#39;s probably the case why you aren&#39;t seeing version conflicts currently\.  h5py is not immune to this situation however, it&#39;s just that you are linking with the latest \[1\.8\.x\] library currently\.  I believe that Andrew is planning on creating wrappers for the new API routines in the 1\.8\.x release, which will complicate your life again\.  Also, if your version of h5py still linked against 1\.8\.x after the 1\.10\.x releases were out and you received a file with format changes from that release, you will have similar problems to the IDL situation\.

Yes, h5py isn't explicitly designed to solve this problem; my advice
as always been "compile against 1.6 if you're seriously worried about
compatibility". H5py has publicly supported almost all of the 1.8
feature set for about a year now (since 1.0); in fact, when compiled
against 1.8 the new API routines (e.g. H5Aiterate2 as opposed to
H5Aiterate1) are usually what's being called. More information about
the mechanics of this is here:

http://h5py.alfven.org/docs/api/index.html

However, this raises a question... does using the "strict" 1.6 API
with HDF5 1.8 (by defining H5_USE_16_API and not calling the new
routines) affect the versions of the structures created? Right now
it's very easy to compile h5py in this mode; in fact, it's the default
if h5py can't discover the version of HDF5 at compile time.

Andrew

Quincey_Koziol · December 4, 2009, 4:08am

Hi Andrew,

···

On Dec 3, 2009, at 10:00 PM, Andrew Collette wrote:

Yes, that's probably the case why you aren't seeing version conflicts currently. h5py is not immune to this situation however, it's just that you are linking with the latest [1.8.x] library currently. I believe that Andrew is planning on creating wrappers for the new API routines in the 1.8.x release, which will complicate your life again. Also, if your version of h5py still linked against 1.8.x after the 1.10.x releases were out and you received a file with format changes from that release, you will have similar problems to the IDL situation.

Yes, h5py isn't explicitly designed to solve this problem; my advice
as always been "compile against 1.6 if you're seriously worried about
compatibility". H5py has publicly supported almost all of the 1.8
feature set for about a year now (since 1.0); in fact, when compiled
against 1.8 the new API routines (e.g. H5Aiterate2 as opposed to
H5Aiterate1) are usually what's being called. More information about
the mechanics of this is here:

http://h5py.alfven.org/docs/api/index.html

However, this raises a question... does using the "strict" 1.6 API
with HDF5 1.8 (by defining H5_USE_16_API and not calling the new
routines) affect the versions of the structures created? Right now
it's very easy to compile h5py in this mode; in fact, it's the default
if h5py can't discover the version of HDF5 at compile time.

In the situation you describe (where an application could be successfully compiled against the 1.6.x version of the library), the files created by the latest 1.8.x release should be identical to files created by the same application linked against the latest 1.6.x release. (I'm including the "latest" caveat here to get over the couple of bugs which generated this discussion)

It's only possible to generate a file with later versions of the file format by using later API routines, or using H5Pset_libver_bounds().

Quincey

newville · December 7, 2009, 9:20pm

Quincey,

Application developers should be prepared for any API call to fail, and handle that situation
correctly.

OK, but how? I've been looking at the docs for the format and API and
all I see is that a negative number is returned on error. I don't see
anywhere how to interpret this value. The Errors that are part of H5E*
stack don't seem to mention mismatches between object versions. Am I
missing something? It's very confusing to hear "yes there are
versions for all objects" followed by "but you can't see them".

Yes, that's probably the case why you aren't seeing version conflicts currently.
h5py is not immune to this situation however, it's just that you are linking with the latest
[1.8.x] library currently. I believe that Andrew is planning on creating wrappers for the new
API routines in the 1.8.x release, which will complicate your life again. Also, if your version
of h5py still linked against 1.8.x after the 1.10.x releases were out and you received a file
with format changes from that release, you will have similar problems to the IDL situation.

That is exactly what scares me. I have been trying to understand if
there are ways to detect version information so that the problems of
"the IDL situation" (in which forward compatibility fails badly) might
be avoided, or at least handled better. I do understand that forward
compatibility is difficult. But why can't "future features" might be
detected so that sensible messages can be given to the end-user by the
applications?

If I write an application today with the 1.8.4 API, what can I do to
avoid problems from someone feeding that application a 1.10.* data
files? Are there tools for the application developer to handle this
problem when reading a file? I don't see these. And yet, there
are two "current" versions of HDF5 and both you and Elena Poumal have
more or less stated that issues with HDF5 versions will happen. I'm
very confused by this: on the one hand interoperability is promised
and stated as a main feature of HDF5, and on the other hand, you can't
really expect this to work without very tight control. Am I alone in
being confused by this?

Do others use HDF5 for such interoperability and archival purposes?
If so, how to they deal with these issues of potential mismatch
between file format and API?

Sorry if this seems crabby. I'm just trying to understand how to use this....

Thanks,

--Matt Newville <newville at cars.uchicago.edu>

Quincey_Koziol · December 8, 2009, 1:35pm

Hi Matt,

Quincey,

Application developers should be prepared for any API call to fail, and handle that situation
correctly.

OK, but how? I've been looking at the docs for the format and API and
all I see is that a negative number is returned on error. I don't see
anywhere how to interpret this value. The Errors that are part of H5E*
stack don't seem to mention mismatches between object versions. Am I
missing something? It's very confusing to hear "yes there are
versions for all objects" followed by "but you can't see them".

You are not supposed to see the file format version information at the API level, it's part of the abstraction layer that the library provides.

If an application wishes to explicitly detect a format version that isn't understood by the HDF5 library, it can register callbacks with the H5E interface to intercept error situations and interrogate the error stack information for the "H5E_VERSION" minor error code, which should be used by the library to indicate a versioning error. Some caution is advised in strongly asserting this to the application user however, since a corrupted file can sometimes appear to be a version from the future to the HDF5 library.

Yes, that's probably the case why you aren't seeing version conflicts currently.
h5py is not immune to this situation however, it's just that you are linking with the latest
[1.8.x] library currently. I believe that Andrew is planning on creating wrappers for the new
API routines in the 1.8.x release, which will complicate your life again. Also, if your version
of h5py still linked against 1.8.x after the 1.10.x releases were out and you received a file
with format changes from that release, you will have similar problems to the IDL situation.

That is exactly what scares me. I have been trying to understand if
there are ways to detect version information so that the problems of
"the IDL situation" (in which forward compatibility fails badly) might
be avoided, or at least handled better. I do understand that forward
compatibility is difficult. But why can't "future features" might be
detected so that sensible messages can be given to the end-user by the
applications?

If I write an application today with the 1.8.4 API, what can I do to
avoid problems from someone feeding that application a 1.10.* data
files?

Write your application so that it checks the error status of every API call and performs sensible actions when one fails. I wouldn't necessarily make the version checking the focus of this error detection, since I think it's rare overall, but rather suggest that it's part of good software design. If an application developer thinks that the HDF5 error information is useful to the user of the application, the hooks are available to provide it.

Are there tools for the application developer to handle this
problem when reading a file? I don't see these. And yet, there
are two "current" versions of HDF5 and both you and Elena Poumal have
more or less stated that issues with HDF5 versions will happen. I'm
very confused by this: on the one hand interoperability is promised
and stated as a main feature of HDF5, and on the other hand, you can't
really expect this to work without very tight control. Am I alone in
being confused by this?

We've wrestled with the forward/backward compatibility issues for a long time (10+ years) and have moved in directions that we think are useful. The "API versioning" macros, the H5Pset_libver_bounds() routine, the full backward compatibility with reading older versions of the file format in each new version of the library, the h5check and h5repack tools, etc. are all ways to help address the problem of adding new library/format features while supporting existing/extinct applications and the files they create. Yes, bugs happen, but we have extensive (and continually expanding) regression test suites that we run daily to try to catch as many as possible. If you've got additional ideas that you think are valuable, we are definitely open to them.

Do others use HDF5 for such interoperability and archival purposes?
If so, how to they deal with these issues of potential mismatch
between file format and API?

Sorry if this seems crabby. I'm just trying to understand how to use this....

Just a little crabby. We're very aware that this can be a complex situation and require some careful thought by application developers, which can be unexpected extra effort, so we're trying to make the path as smooth as possible.

Quincey

···

On Dec 7, 2009, at 3:20 PM, Matt Newville wrote: