I just came around an interesting issue.
I implemented the writing of HDF files on an embedded system. The amount of functionality I implemented is significant less than the HDF lib offers. So it is just tailored to my needs. I implemented everything on base of the HDF 3.0 file spec. One point of my tailoring was to optimize the file size. Therefore, I write every internal block in the HDF files aligned byte-by-byte to the next - or padded to the address alignment if it is requested by the HDF file specification. The HDF files generated by HDFview or Matlab have plenty of space in-between the internal blocks. Sometimes a few hundred bytes. As far as I read from the HDF file specification this 'extended padding' is not defined at all - not even recommended.
However, this 'extended padding' that is performed by the HDF lib leads to a behavior that I would consider as an incompatibility to itself. To demonstrate this I attached two HDF files to this email. The first (sizeoptimized.h5) is generated by my embedded software and is optimized concerning the file size. It contains three compounds with each of them has 2 elements. You should be able to open that file in HDFview or similar tools and read all its contents.
The second file (sizeoptimizedextended.h5) is generated by HDFview by adding a fourth compound after the sizeoptimized.h5 file was opened in HDFview. You can see that the file is partly corrupted. The reason for this is that HDFview (and therefore the HDF lib I guess) is not really taking care about the position of the internal blocks of a file that it is writing to. It seems to me it has some internal mapping of those blocks. This mapping gets applied even if it will collide, and therefore corrupt, the existing blocks.
If my observation is correct I think the HDF lib will need a bugfix or the HDF file spec will need a description of how the internal blocks are allowed to be positioned within a HDF file.
I forgot to mention that I tried to use the HDF lib sources and compile it to my system. However, I quit after a couple of days because the way the sources are written are not suitable at all to adopt them to an embedded system that runs a simplified file system and a real-time operating system - and all of it has to fit into a few hundred kilobytes.
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Krug, Markus
Sent: Tuesday, September 05, 2017 2:57 AM
To: HDF Users Discussion List
Subject: [Hdf-forum] HDF lib incompatible with HDF file spec?
Dear all,
I just came around an interesting issue.
I implemented the writing of HDF files on an embedded system. The amount of functionality I implemented is significant less than the HDF lib offers. So it is just tailored to my needs. I implemented everything on base of the HDF 3.0 file spec. One point of my tailoring was to optimize the file size. Therefore, I write every internal block in the HDF files aligned byte-by-byte to the next - or padded to the address alignment if it is requested by the HDF file specification. The HDF files generated by HDFview or Matlab have plenty of space in-between the internal blocks. Sometimes a few hundred bytes. As far as I read from the HDF file specification this 'extended padding' is not defined at all - not even recommended.
However, this 'extended padding' that is performed by the HDF lib leads to a behavior that I would consider as an incompatibility to itself. To demonstrate this I attached two HDF files to this email. The first (sizeoptimized.h5) is generated by my embedded software and is optimized concerning the file size. It contains three compounds with each of them has 2 elements. You should be able to open that file in HDFview or similar tools and read all its contents.
The second file (sizeoptimizedextended.h5) is generated by HDFview by adding a fourth compound after the sizeoptimized.h5 file was opened in HDFview. You can see that the file is partly corrupted. The reason for this is that HDFview (and therefore the HDF lib I guess) is not really taking care about the position of the internal blocks of a file that it is writing to. It seems to me it has some internal mapping of those blocks. This mapping gets applied even if it will collide, and therefore corrupt, the existing blocks.
If my observation is correct I think the HDF lib will need a bugfix or the HDF file spec will need a description of how the internal blocks are allowed to be positioned within a HDF file.
I forgot to mention that I tried to use the HDF lib sources and compile it to my system. However, I quit after a couple of days because the way the sources are written are not suitable at all to adopt them to an embedded system that runs a simplified file system and a real-time operating system - and all of it has to fit into a few hundred kilobytes.
Just letting you know that bug HDFFV-10300 was entered for this issue.
5
We will investigate it and get back to you on this.
Sorry to jump in, but this time I have to ask: Is the bug tracker
where these "HDFFV" bugs are filed available somewhere? I know I've
seen some discussions in the past about opening up the HDF5
development process. Has there been any movements on this yet?
I'm quite interested in following the triaging/work/discussions around
this bug, but if there's no bug tracker where I can subscribe to the
bug activity, like I can with most other open source projects, I don't
see how I can do that. Whenever one of these mail threads end with a
"HDFFV-XXX has been entered", it's as if the issue disappears into a
black hole, only to come out again when the problem is already solved.
I find that sad, because no-one outside of the HDF5 Group can really
learn anything from this
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of
Krug, Markus
Sent: Tuesday, September 05, 2017 2:57 AM
To: HDF Users Discussion List
Subject: [Hdf-forum] HDF lib incompatible with HDF file spec?
Dear all,
I just came around an interesting issue.
I implemented the writing of HDF files on an embedded system. The amount of
functionality I implemented is significant less than the HDF lib offers. So
it is just tailored to my needs. I implemented everything on base of the HDF
3.0 file spec. One point of my tailoring was to optimize the file size.
Therefore, I write every internal block in the HDF files aligned
byte-by-byte to the next – or padded to the address alignment if it is
requested by the HDF file specification. The HDF files generated by HDFview
or Matlab have plenty of space in-between the internal blocks. Sometimes a
few hundred bytes. As far as I read from the HDF file specification this
‘extended padding’ is not defined at all – not even recommended.
However, this ‘extended padding’ that is performed by the HDF lib leads to a
behavior that I would consider as an incompatibility to itself. To
demonstrate this I attached two HDF files to this email. The first
(sizeoptimized.h5) is generated by my embedded software and is optimized
concerning the file size. It contains three compounds with each of them has
2 elements. You should be able to open that file in HDFview or similar tools
and read all its contents.
The second file (sizeoptimizedextended.h5) is generated by HDFview by adding
a fourth compound after the sizeoptimized.h5 file was opened in HDFview. You
can see that the file is partly corrupted. The reason for this is that
HDFview (and therefore the HDF lib I guess) is not really taking care about
the position of the internal blocks of a file that it is writing to. It
seems to me it has some internal mapping of those blocks. This mapping gets
applied even if it will collide, and therefore corrupt, the existing blocks.
If my observation is correct I think the HDF lib will need a bugfix or the
HDF file spec will need a description of how the internal blocks are allowed
to be positioned within a HDF file.
I forgot to mention that I tried to use the HDF lib sources and compile it
to my system. However, I quit after a couple of days because the way the
sources are written are not suitable at all to adopt them to an embedded
system that runs a simplified file system and a real-time operating system –
and all of it has to fit into a few hundred kilobytes.
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Krug, Markus
Sent: Tuesday, September 05, 2017 2:57 AM
To: HDF Users Discussion List
Subject: [Hdf-forum] HDF lib incompatible with HDF file spec?
Dear all,
I just came around an interesting issue.
I implemented the writing of HDF files on an embedded system. The amount of functionality I implemented is significant less than the HDF lib offers. So it is just tailored to my needs. I implemented everything on base of the HDF 3.0 file spec. One point of my tailoring was to optimize the file size. Therefore, I write every internal block in the HDF files aligned byte-by-byte to the next - or padded to the address alignment if it is requested by the HDF file specification. The HDF files generated by HDFview or Matlab have plenty of space in-between the internal blocks. Sometimes a few hundred bytes. As far as I read from the HDF file specification this 'extended padding' is not defined at all - not even recommended.
However, this 'extended padding' that is performed by the HDF lib leads to a behavior that I would consider as an incompatibility to itself. To demonstrate this I attached two HDF files to this email. The first (sizeoptimized.h5) is generated by my embedded software and is optimized concerning the file size. It contains three compounds with each of them has 2 elements. You should be able to open that file in HDFview or similar tools and read all its contents.
The second file (sizeoptimizedextended.h5) is generated by HDFview by adding a fourth compound after the sizeoptimized.h5 file was opened in HDFview. You can see that the file is partly corrupted. The reason for this is that HDFview (and therefore the HDF lib I guess) is not really taking care about the position of the internal blocks of a file that it is writing to. It seems to me it has some internal mapping of those blocks. This mapping gets applied even if it will collide, and therefore corrupt, the existing blocks.
If my observation is correct I think the HDF lib will need a bugfix or the HDF file spec will need a description of how the internal blocks are allowed to be positioned within a HDF file.
I forgot to mention that I tried to use the HDF lib sources and compile it to my system. However, I quit after a couple of days because the way the sources are written are not suitable at all to adopt them to an embedded system that runs a simplified file system and a real-time operating system - and all of it has to fit into a few hundred kilobytes.
Our bug tracking system is not currently open to the public. I'm not sure when it will be, but please
feel free to contact the helpdesk (help@hdfgroup.org) if you ever want to check on the status of a bug report.
-Barbara
help@hdfgroup.org
···
-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Elvis Stansvik
Sent: Monday, October 02, 2017 12:29 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] HDF lib incompatible with HDF file spec?
2017-10-02 18:56 GMT+02:00 Barbara Jones <bljones@hdfgroup.org>:
Hi Markus,
Just letting you know that bug HDFFV-10300 was entered for this issue.
5
We will investigate it and get back to you on this.
Sorry to jump in, but this time I have to ask: Is the bug tracker where these "HDFFV" bugs are filed available somewhere? I know I've seen some discussions in the past about opening up the HDF5 development process. Has there been any movements on this yet?
I'm quite interested in following the triaging/work/discussions around this bug, but if there's no bug tracker where I can subscribe to the bug activity, like I can with most other open source projects, I don't see how I can do that. Whenever one of these mail threads end with a "HDFFV-XXX has been entered", it's as if the issue disappears into a black hole, only to come out again when the problem is already solved.
I find that sad, because no-one outside of the HDF5 Group can really learn anything from this
Elvis
Thanks!
-Barbara
help@hdfgroup.org
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On
Behalf Of Krug, Markus
Sent: Tuesday, September 05, 2017 2:57 AM
To: HDF Users Discussion List
Subject: [Hdf-forum] HDF lib incompatible with HDF file spec?
Dear all,
I just came around an interesting issue.
I implemented the writing of HDF files on an embedded system. The
amount of functionality I implemented is significant less than the HDF
lib offers. So it is just tailored to my needs. I implemented
everything on base of the HDF
3.0 file spec. One point of my tailoring was to optimize the file size.
Therefore, I write every internal block in the HDF files aligned
byte-by-byte to the next – or padded to the address alignment if it is
requested by the HDF file specification. The HDF files generated by
HDFview or Matlab have plenty of space in-between the internal blocks.
Sometimes a few hundred bytes. As far as I read from the HDF file
specification this ‘extended padding’ is not defined at all – not even recommended.
However, this ‘extended padding’ that is performed by the HDF lib
leads to a behavior that I would consider as an incompatibility to
itself. To demonstrate this I attached two HDF files to this email.
The first
(sizeoptimized.h5) is generated by my embedded software and is
optimized concerning the file size. It contains three compounds with
each of them has
2 elements. You should be able to open that file in HDFview or similar
tools and read all its contents.
The second file (sizeoptimizedextended.h5) is generated by HDFview by
adding a fourth compound after the sizeoptimized.h5 file was opened in
HDFview. You can see that the file is partly corrupted. The reason for
this is that HDFview (and therefore the HDF lib I guess) is not really
taking care about the position of the internal blocks of a file that
it is writing to. It seems to me it has some internal mapping of those
blocks. This mapping gets applied even if it will collide, and therefore corrupt, the existing blocks.
If my observation is correct I think the HDF lib will need a bugfix or
the HDF file spec will need a description of how the internal blocks
are allowed to be positioned within a HDF file.
I forgot to mention that I tried to use the HDF lib sources and
compile it to my system. However, I quit after a couple of days
because the way the sources are written are not suitable at all to
adopt them to an embedded system that runs a simplified file system
and a real-time operating system – and all of it has to fit into a few hundred kilobytes.
Our bug tracking system is not currently open to the public. I'm not sure when it will be, but please
feel free to contact the helpdesk (help@hdfgroup.org) if you ever want to check on the status of a bug report.
Alright, glad to know that it's at least still planned (?).
My frustration wasn't simply about the status of a bug, but the lack
of transparency in the process leading to its conclusion. An open bug
tracker is much more than a means of checking on the status of a bug,
it provides a place to get insight into and collaborate on the
solution.
-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Elvis Stansvik
Sent: Monday, October 02, 2017 12:29 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] HDF lib incompatible with HDF file spec?
Just letting you know that bug HDFFV-10300 was entered for this issue.
5
We will investigate it and get back to you on this.
Sorry to jump in, but this time I have to ask: Is the bug tracker where these "HDFFV" bugs are filed available somewhere? I know I've seen some discussions in the past about opening up the HDF5 development process. Has there been any movements on this yet?
I'm quite interested in following the triaging/work/discussions around this bug, but if there's no bug tracker where I can subscribe to the bug activity, like I can with most other open source projects, I don't see how I can do that. Whenever one of these mail threads end with a "HDFFV-XXX has been entered", it's as if the issue disappears into a black hole, only to come out again when the problem is already solved.
I find that sad, because no-one outside of the HDF5 Group can really learn anything from this
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On
Behalf Of Krug, Markus
Sent: Tuesday, September 05, 2017 2:57 AM
To: HDF Users Discussion List
Subject: [Hdf-forum] HDF lib incompatible with HDF file spec?
Dear all,
I just came around an interesting issue.
I implemented the writing of HDF files on an embedded system. The
amount of functionality I implemented is significant less than the HDF
lib offers. So it is just tailored to my needs. I implemented
everything on base of the HDF
3.0 file spec. One point of my tailoring was to optimize the file size.
Therefore, I write every internal block in the HDF files aligned
byte-by-byte to the next – or padded to the address alignment if it is
requested by the HDF file specification. The HDF files generated by
HDFview or Matlab have plenty of space in-between the internal blocks.
Sometimes a few hundred bytes. As far as I read from the HDF file
specification this ‘extended padding’ is not defined at all – not even recommended.
However, this ‘extended padding’ that is performed by the HDF lib
leads to a behavior that I would consider as an incompatibility to
itself. To demonstrate this I attached two HDF files to this email.
The first
(sizeoptimized.h5) is generated by my embedded software and is
optimized concerning the file size. It contains three compounds with
each of them has
2 elements. You should be able to open that file in HDFview or similar
tools and read all its contents.
The second file (sizeoptimizedextended.h5) is generated by HDFview by
adding a fourth compound after the sizeoptimized.h5 file was opened in
HDFview. You can see that the file is partly corrupted. The reason for
this is that HDFview (and therefore the HDF lib I guess) is not really
taking care about the position of the internal blocks of a file that
it is writing to. It seems to me it has some internal mapping of those
blocks. This mapping gets applied even if it will collide, and therefore corrupt, the existing blocks.
If my observation is correct I think the HDF lib will need a bugfix or
the HDF file spec will need a description of how the internal blocks
are allowed to be positioned within a HDF file.
I forgot to mention that I tried to use the HDF lib sources and
compile it to my system. However, I quit after a couple of days
because the way the sources are written are not suitable at all to
adopt them to an embedded system that runs a simplified file system
and a real-time operating system – and all of it has to fit into a few hundred kilobytes.
