Linked block in HDF4 files

Hdf-forum archives
1

Hi all,

To give some background context, I'm working on a Java project that deals
with HDF files. I used to use the Java-interface to the native HDF library,
however, I had some issues with it and I wanted to use a pure Java
library. I didn't find any pure Java HDF libraries, so I decided to make
my own. It's going very fine and it fits very well in my project. It's not
complete yet, but I'm extending it bit-by-bit. All my work is open source.
Currently, it's part of a bigger SpatialHadoop project [
https://github.com/aseldawy/spatialhadoop2]. However, I'm planning to ship
it is a standalone library which will be available for the community.

Currently, I'm working on the Linked Block feature of HDF files. However,
I'm having problems with this file
http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.02.18/MOD13Q1.A2000049.h10v11.005.2006270195941.hdf
The file opens fine in HDF viewer, which means that the file is correct.
I'm following the HDF specs at
http://www.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Linked block elements are described in Section 10.3 on page 149 (160 in the
PDF). First, there seems to be a problem in the documentation as it says
that EXT_LINKED constant is 32-bits. This seems to be error because other
extension identifiers are 16-bits (e.g., see SPECIAL_EXT in 10.4).
Anyway, I fixed it myself and worked with a 16-bit identifier. However, I
started to see some problems and I don't know if my fix of 32-bits to
16-bits was right or wrong.
The file I mentioned above has an extended data block with tagID 1963
(DFTAG_VS) and reference no 49. The data of this object is located in
offset 38215 and length 16 bytes. When I parse this block according to the
specs, I get the following numbers which are inconsistent.
Length: 57600
first_len: 4096
blk_len: 16
num_blk: 131072
Now these numbers do not make sense because if the length of the first
block is 4096 and then we have 131072 blocks, each of 16-bytes length, the
total length would be 2101248 which is much larger than 57600.
I'm having more troubles when parsing the linked block table but let's
resolve this issue first as the later might be related.

2

Hi Ahmed,

I'm looking into this to find out what the problem is.

?Binh-Minh

···

________________________________
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Ahmed Eldawy <aseldawy+hdf@gmail.com>
Sent: Thursday, June 18, 2015 4:04 PM
To: hdf-forum
Subject: [Hdf-forum] Linked block in HDF4 files

Hi all,

To give some background context, I'm working on a Java project that deals with HDF files. I used to use the Java-interface to the native HDF library, however, I had some issues with it and I wanted to use a pure Java library. I didn't find any pure Java HDF libraries, so I decided to make my own. It's going very fine and it fits very well in my project. It's not complete yet, but I'm extending it bit-by-bit. All my work is open source. Currently, it's part of a bigger SpatialHadoop project [https://github.com/aseldawy/spatialhadoop2]. However, I'm planning to ship it is a standalone library which will be available for the community.

Currently, I'm working on the Linked Block feature of HDF files. However, I'm having problems with this file
http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.02.18/MOD13Q1.A2000049.h10v11.005.2006270195941.hdf
The file opens fine in HDF viewer, which means that the file is correct. I'm following the HDF specs at
http://www.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Linked block elements are described in Section 10.3 on page 149 (160 in the PDF). First, there seems to be a problem in the documentation as it says that EXT_LINKED constant is 32-bits. This seems to be error because other extension identifiers are 16-bits (e.g., see SPECIAL_EXT in 10.4).
Anyway, I fixed it myself and worked with a 16-bit identifier. However, I started to see some problems and I don't know if my fix of 32-bits to 16-bits was right or wrong.
The file I mentioned above has an extended data block with tagID 1963 (DFTAG_VS) and reference no 49. The data of this object is located in offset 38215 and length 16 bytes. When I parse this block according to the specs, I get the following numbers which are inconsistent.
Length: 57600
first_len: 4096
blk_len: 16
num_blk: 131072
Now these numbers do not make sense because if the length of the first block is 4096 and then we have 131072 blocks, each of 16-bytes length, the total length would be 2101248 which is much larger than 57600.
I'm having more troubles when parsing the linked block table but let's resolve this issue first as the later might be related.

3

Hi Binh-Minh Ribler,
  Thanks for your response. I've been looking into the issue myself and I
think I found the problem. I had to check the source code of the HDF
library and I found some useful comments in the file hblocks.c. According
to this file, the correct header format is
   ext_tag_desc - SPECIAL_LINKED(16 bit constant), identifies this as a
linked block description record
   elem_tot_len - Length of the entire element(32 bit field)
   blk_length - Length of successive data blocks(32 bit field) after
first block,
   first block is calculated.
   num_blk - Number of blocks per block table(32 bit field)
   link_ref - Reference number of the first block table(16 bit field)

As I mentioned earlier, the ext_tag_desc is 16-bits only and the constant
is SPECIAL_LINKED not EXT_LINKED. In addition, the value first_blk_len is
not actually stored in the header. It is calculated based on number of
blocks, the size of each block, and the total element length.
After I fixed these issues, the numbers started to make more sense. Do you
know how I can reach out to the people who write the documentation so that
they can solve these problems? It seems that the documentation is outdated.

···

On Thu, Jun 18, 2015 at 3:04 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com> wrote:

Hi all,

To give some background context, I'm working on a Java project that deals
with HDF files. I used to use the Java-interface to the native HDF library,
however, I had some issues with it and I wanted to use a pure Java
library. I didn't find any pure Java HDF libraries, so I decided to make
my own. It's going very fine and it fits very well in my project. It's not
complete yet, but I'm extending it bit-by-bit. All my work is open source.
Currently, it's part of a bigger SpatialHadoop project [
https://github.com/aseldawy/spatialhadoop2]. However, I'm planning to
ship it is a standalone library which will be available for the community.

Currently, I'm working on the Linked Block feature of HDF files. However,
I'm having problems with this file

http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.02.18/MOD13Q1.A2000049.h10v11.005.2006270195941.hdf
The file opens fine in HDF viewer, which means that the file is correct.
I'm following the HDF specs at
http://www.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Linked block elements are described in Section 10.3 on page 149 (160 in
the PDF). First, there seems to be a problem in the documentation as it
says that EXT_LINKED constant is 32-bits. This seems to be error because
other extension identifiers are 16-bits (e.g., see SPECIAL_EXT in 10.4).
Anyway, I fixed it myself and worked with a 16-bit identifier. However, I
started to see some problems and I don't know if my fix of 32-bits to
16-bits was right or wrong.
The file I mentioned above has an extended data block with tagID 1963
(DFTAG_VS) and reference no 49. The data of this object is located in
offset 38215 and length 16 bytes. When I parse this block according to the
specs, I get the following numbers which are inconsistent.
Length: 57600
first_len: 4096
blk_len: 16
num_blk: 131072
Now these numbers do not make sense because if the length of the first
block is 4096 and then we have 131072 blocks, each of 16-bytes length, the
total length would be 2101248 which is much larger than 57600.
I'm having more troubles when parsing the linked block table but let's
resolve this issue first as the later might be related.

4

Hi Ahmed,

You are correct that the Spec is outdated, particularly Chapters 9-11 and Appendix A. I'm working on updating it. In the mean time, please refer to the User's Guide and Reference Manual for more up-to-date information, where applicable.

https://www.hdfgroup.org/release4/doc/UsrGuide_html/UG_Top.html

https://www.hdfgroup.org/release4/doc/RefMan_html/RM_Top.html

In the coming release, in Feb, the Specs will be updated.

Thank you for the info!

​Binh-Minh

···

________________________________
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Ahmed Eldawy <aseldawy+hdf@gmail.com>
Sent: Monday, June 29, 2015 12:50 AM
To: hdf-forum
Subject: Re: [Hdf-forum] Linked block in HDF4 files

Hi Binh-Minh Ribler,
  Thanks for your response. I've been looking into the issue myself and I think I found the problem. I had to check the source code of the HDF library and I found some useful comments in the file hblocks.c. According to this file, the correct header format is
   ext_tag_desc - SPECIAL_LINKED(16 bit constant), identifies this as a linked block description record
   elem_tot_len - Length of the entire element(32 bit field)
   blk_length - Length of successive data blocks(32 bit field) after first block,
   first block is calculated.
   num_blk - Number of blocks per block table(32 bit field)
   link_ref - Reference number of the first block table(16 bit field)

As I mentioned earlier, the ext_tag_desc is 16-bits only and the constant is SPECIAL_LINKED not EXT_LINKED. In addition, the value first_blk_len is not actually stored in the header. It is calculated based on number of blocks, the size of each block, and the total element length.
After I fixed these issues, the numbers started to make more sense. Do you know how I can reach out to the people who write the documentation so that they can solve these problems? It seems that the documentation is outdated.

On Thu, Jun 18, 2015 at 3:04 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com<mailto:aseldawy+hdf@gmail.com>> wrote:
Hi all,

To give some background context, I'm working on a Java project that deals with HDF files. I used to use the Java-interface to the native HDF library, however, I had some issues with it and I wanted to use a pure Java library. I didn't find any pure Java HDF libraries, so I decided to make my own. It's going very fine and it fits very well in my project. It's not complete yet, but I'm extending it bit-by-bit. All my work is open source. Currently, it's part of a bigger SpatialHadoop project [https://github.com/aseldawy/spatialhadoop2]. However, I'm planning to ship it is a standalone library which will be available for the community.

Currently, I'm working on the Linked Block feature of HDF files. However, I'm having problems with this file
http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.02.18/MOD13Q1.A2000049.h10v11.005.2006270195941.hdf
The file opens fine in HDF viewer, which means that the file is correct. I'm following the HDF specs at
http://www.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Linked block elements are described in Section 10.3 on page 149 (160 in the PDF). First, there seems to be a problem in the documentation as it says that EXT_LINKED constant is 32-bits. This seems to be error because other extension identifiers are 16-bits (e.g., see SPECIAL_EXT in 10.4).
Anyway, I fixed it myself and worked with a 16-bit identifier. However, I started to see some problems and I don't know if my fix of 32-bits to 16-bits was right or wrong.
The file I mentioned above has an extended data block with tagID 1963 (DFTAG_VS) and reference no 49. The data of this object is located in offset 38215 and length 16 bytes. When I parse this block according to the specs, I get the following numbers which are inconsistent.
Length: 57600
first_len: 4096
blk_len: 16
num_blk: 131072
Now these numbers do not make sense because if the length of the first block is 4096 and then we have 131072 blocks, each of 16-bytes length, the total length would be 2101248 which is much larger than 57600.
I'm having more troubles when parsing the linked block table but let's resolve this issue first as the later might be related.

5

Hi Ahmed,

I think the HDF4 Mapping tool that I mentioned to you previously:
    http://www.hdfgroup.org/projects/h4map/
may be helpful when you need to look at the raw data in the file. Using this tool, you will be able to see the structure in the file more easily.

As for the Specs, please continue sending problems that you find and we will enter them in our JIRA database and will update the document for the 4.2.12 release, according to their priority.

Thank you,

Binh-Minh

···

________________________________

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Ahmed Eldawy <aseldawy+hdf@gmail.com>
Sent: Monday, June 29, 2015 12:50 AM
To: hdf-forum
Subject: Re: [Hdf-forum] Linked block in HDF4 files

Hi Binh-Minh Ribler,
  Thanks for your response. I've been looking into the issue myself and I think I found the problem. I had to check the source code of the HDF library and I found some useful comments in the file hblocks.c. According to this file, the correct header format is
   ext_tag_desc - SPECIAL_LINKED(16 bit constant), identifies this as a linked block description record
   elem_tot_len - Length of the entire element(32 bit field)
   blk_length - Length of successive data blocks(32 bit field) after first block,
   first block is calculated.
   num_blk - Number of blocks per block table(32 bit field)
   link_ref - Reference number of the first block table(16 bit field)

As I mentioned earlier, the ext_tag_desc is 16-bits only and the constant is SPECIAL_LINKED not EXT_LINKED. In addition, the value first_blk_len is not actually stored in the header. It is calculated based on number of blocks, the size of each block, and the total element length.
After I fixed these issues, the numbers started to make more sense. Do you know how I can reach out to the people who write the documentation so that they can solve these problems? It seems that the documentation is outdated.

On Thu, Jun 18, 2015 at 3:04 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com<mailto:aseldawy+hdf@gmail.com>> wrote:
Hi all,

To give some background context, I'm working on a Java project that deals with HDF files. I used to use the Java-interface to the native HDF library, however, I had some issues with it and I wanted to use a pure Java library. I didn't find any pure Java HDF libraries, so I decided to make my own. It's going very fine and it fits very well in my project. It's not complete yet, but I'm extending it bit-by-bit. All my work is open source. Currently, it's part of a bigger SpatialHadoop project [https://github.com/aseldawy/spatialhadoop2]. However, I'm planning to ship it is a standalone library which will be available for the community.

Currently, I'm working on the Linked Block feature of HDF files. However, I'm having problems with this file
http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.02.18/MOD13Q1.A2000049.h10v11.005.2006270195941.hdf
The file opens fine in HDF viewer, which means that the file is correct. I'm following the HDF specs at
http://www.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Linked block elements are described in Section 10.3 on page 149 (160 in the PDF). First, there seems to be a problem in the documentation as it says that EXT_LINKED constant is 32-bits. This seems to be error because other extension identifiers are 16-bits (e.g., see SPECIAL_EXT in 10.4).
Anyway, I fixed it myself and worked with a 16-bit identifier. However, I started to see some problems and I don't know if my fix of 32-bits to 16-bits was right or wrong.
The file I mentioned above has an extended data block with tagID 1963 (DFTAG_VS) and reference no 49. The data of this object is located in offset 38215 and length 16 bytes. When I parse this block according to the specs, I get the following numbers which are inconsistent.
Length: 57600
first_len: 4096
blk_len: 16
num_blk: 131072
Now these numbers do not make sense because if the length of the first block is 4096 and then we have 131072 blocks, each of 16-bytes length, the total length would be 2101248 which is much larger than 57600.
I'm having more troubles when parsing the linked block table but let's resolve this issue first as the later might be related.

6

I've seen your previous comment about the HDF4 Mapping Tool. However, I
couldn't find any instructions on how to install or use it. If the
generated XML file looks like this
http://e4ftl01.cr.usgs.gov/MOLA/MYD11A1.005/2002.07.08/MYD11A1.A2002189.h18v06.005.2007216142720.hdf.xml
then I don't think it will be of much help to me.

···

On Sun, Jun 28, 2015 at 11:50 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com> wrote:

Hi Binh-Minh Ribler,
  Thanks for your response. I've been looking into the issue myself and I
think I found the problem. I had to check the source code of the HDF
library and I found some useful comments in the file hblocks.c. According
to this file, the correct header format is
   ext_tag_desc - SPECIAL_LINKED(16 bit constant), identifies this as a
linked block description record
   elem_tot_len - Length of the entire element(32 bit field)
   blk_length - Length of successive data blocks(32 bit field) after
first block,
   first block is calculated.
   num_blk - Number of blocks per block table(32 bit field)
   link_ref - Reference number of the first block table(16 bit field)

As I mentioned earlier, the ext_tag_desc is 16-bits only and the constant
is SPECIAL_LINKED not EXT_LINKED. In addition, the value first_blk_len is
not actually stored in the header. It is calculated based on number of
blocks, the size of each block, and the total element length.
After I fixed these issues, the numbers started to make more sense. Do you
know how I can reach out to the people who write the documentation so that
they can solve these problems? It seems that the documentation is outdated.

On Thu, Jun 18, 2015 at 3:04 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com> > wrote:

Hi all,

To give some background context, I'm working on a Java project that
deals with HDF files. I used to use the Java-interface to the native HDF
library, however, I had some issues with it and I wanted to use a pure Java
library. I didn't find any pure Java HDF libraries, so I decided to make
my own. It's going very fine and it fits very well in my project. It's not
complete yet, but I'm extending it bit-by-bit. All my work is open source.
Currently, it's part of a bigger SpatialHadoop project [
https://github.com/aseldawy/spatialhadoop2]. However, I'm planning to
ship it is a standalone library which will be available for the community.

Currently, I'm working on the Linked Block feature of HDF files.
However, I'm having problems with this file

http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.02.18/MOD13Q1.A2000049.h10v11.005.2006270195941.hdf
The file opens fine in HDF viewer, which means that the file is correct.
I'm following the HDF specs at
http://www.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Linked block elements are described in Section 10.3 on page 149 (160 in
the PDF). First, there seems to be a problem in the documentation as it
says that EXT_LINKED constant is 32-bits. This seems to be error because
other extension identifiers are 16-bits (e.g., see SPECIAL_EXT in 10.4).
Anyway, I fixed it myself and worked with a 16-bit identifier. However, I
started to see some problems and I don't know if my fix of 32-bits to
16-bits was right or wrong.
The file I mentioned above has an extended data block with tagID 1963
(DFTAG_VS) and reference no 49. The data of this object is located in
offset 38215 and length 16 bytes. When I parse this block according to the
specs, I get the following numbers which are inconsistent.
Length: 57600
first_len: 4096
blk_len: 16
num_blk: 131072
Now these numbers do not make sense because if the length of the first
block is 4096 and then we have 131072 blocks, each of 16-bytes length, the
total length would be 2101248 which is much larger than 57600.
I'm having more troubles when parsing the linked block table but let's
resolve this issue first as the later might be related.

7

Hi Ahmed,

Did you get the tool from the hdfgroup.org website as h4map.tar.gz? If so, please follow the brief instruction below to install the tool:

- Unpack the package using tar –xvf h4map.tar.gz.

- Change directory to h4map and run BUILD_AND_INSTALL shell script. This also builds the hdf4 library for h4map.

- Look for the h4mapwriter binary under ./install/bin directory.

Please note that the writer was implemented and tested for CentOS5 32-bit and 64-bit Linux platforms only.

Thanks,

​Binh-Minh

···

________________________________
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Ahmed Eldawy <aseldawy+hdf@gmail.com>
Sent: Thursday, July 02, 2015 5:52 AM
To: hdf-forum
Subject: Re: [Hdf-forum] Linked block in HDF4 files

I've seen your previous comment about the HDF4 Mapping Tool. However, I couldn't find any instructions on how to install or use it. If the generated XML file looks like this
http://e4ftl01.cr.usgs.gov/MOLA/MYD11A1.005/2002.07.08/MYD11A1.A2002189.h18v06.005.2007216142720.hdf.xml
then I don't think it will be of much help to me.

On Sun, Jun 28, 2015 at 11:50 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com<mailto:aseldawy+hdf@gmail.com>> wrote:
Hi Binh-Minh Ribler,
  Thanks for your response. I've been looking into the issue myself and I think I found the problem. I had to check the source code of the HDF library and I found some useful comments in the file hblocks.c. According to this file, the correct header format is
   ext_tag_desc - SPECIAL_LINKED(16 bit constant), identifies this as a linked block description record
   elem_tot_len - Length of the entire element(32 bit field)
   blk_length - Length of successive data blocks(32 bit field) after first block,
   first block is calculated.
   num_blk - Number of blocks per block table(32 bit field)
   link_ref - Reference number of the first block table(16 bit field)

As I mentioned earlier, the ext_tag_desc is 16-bits only and the constant is SPECIAL_LINKED not EXT_LINKED. In addition, the value first_blk_len is not actually stored in the header. It is calculated based on number of blocks, the size of each block, and the total element length.
After I fixed these issues, the numbers started to make more sense. Do you know how I can reach out to the people who write the documentation so that they can solve these problems? It seems that the documentation is outdated.

On Thu, Jun 18, 2015 at 3:04 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com<mailto:aseldawy+hdf@gmail.com>> wrote:
Hi all,

To give some background context, I'm working on a Java project that deals with HDF files. I used to use the Java-interface to the native HDF library, however, I had some issues with it and I wanted to use a pure Java library. I didn't find any pure Java HDF libraries, so I decided to make my own. It's going very fine and it fits very well in my project. It's not complete yet, but I'm extending it bit-by-bit. All my work is open source. Currently, it's part of a bigger SpatialHadoop project [https://github.com/aseldawy/spatialhadoop2]. However, I'm planning to ship it is a standalone library which will be available for the community.

Currently, I'm working on the Linked Block feature of HDF files. However, I'm having problems with this file
http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.02.18/MOD13Q1.A2000049.h10v11.005.2006270195941.hdf
The file opens fine in HDF viewer, which means that the file is correct. I'm following the HDF specs at
http://www.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Linked block elements are described in Section 10.3 on page 149 (160 in the PDF). First, there seems to be a problem in the documentation as it says that EXT_LINKED constant is 32-bits. This seems to be error because other extension identifiers are 16-bits (e.g., see SPECIAL_EXT in 10.4).
Anyway, I fixed it myself and worked with a 16-bit identifier. However, I started to see some problems and I don't know if my fix of 32-bits to 16-bits was right or wrong.
The file I mentioned above has an extended data block with tagID 1963 (DFTAG_VS) and reference no 49. The data of this object is located in offset 38215 and length 16 bytes. When I parse this block according to the specs, I get the following numbers which are inconsistent.
Length: 57600
first_len: 4096
blk_len: 16
num_blk: 131072
Now these numbers do not make sense because if the length of the first block is 4096 and then we have 131072 blocks, each of 16-bytes length, the total length would be 2101248 which is much larger than 57600.
I'm having more troubles when parsing the linked block table but let's resolve this issue first as the later might be related.

8

Hello again, Ahmed,

The h4map displays the offset and length of the data linked-blocks similar to this sample:

                            <h4:byteStream offset="2905156" nBytes="196608"/>

                            <h4:byteStream offset="3573137" nBytes="131072"/>

                            <h4:byteStream offset="4113809" nBytes="131072"/>

It displays the offset and length of some of the data chunks in your file like this:

                            <h4:byteStream offset="37500" nBytes="35" chunkPositionInArray="[0,0]"/>

                            <h4:byteStream offset="42377" nBytes="35" chunkPositionInArray="[1,0]"/>

                            <h4:byteStream offset="88582" nBytes="35" chunkPositionInArray="[2,0]"/>

                            <h4:byteStream offset="89181" nBytes="35" chunkPositionInArray="[3,0]"/>

Do you think this kind of information is useful in your project?
​Binh-Minh

···

________________________________
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Ahmed Eldawy <aseldawy+hdf@gmail.com>
Sent: Thursday, July 02, 2015 5:52 AM
To: hdf-forum
Subject: Re: [Hdf-forum] Linked block in HDF4 files

I've seen your previous comment about the HDF4 Mapping Tool. However, I couldn't find any instructions on how to install or use it. If the generated XML file looks like this
http://e4ftl01.cr.usgs.gov/MOLA/MYD11A1.005/2002.07.08/MYD11A1.A2002189.h18v06.005.2007216142720.hdf.xml
then I don't think it will be of much help to me.

On Sun, Jun 28, 2015 at 11:50 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com<mailto:aseldawy+hdf@gmail.com>> wrote:
Hi Binh-Minh Ribler,
  Thanks for your response. I've been looking into the issue myself and I think I found the problem. I had to check the source code of the HDF library and I found some useful comments in the file hblocks.c. According to this file, the correct header format is
   ext_tag_desc - SPECIAL_LINKED(16 bit constant), identifies this as a linked block description record
   elem_tot_len - Length of the entire element(32 bit field)
   blk_length - Length of successive data blocks(32 bit field) after first block,
   first block is calculated.
   num_blk - Number of blocks per block table(32 bit field)
   link_ref - Reference number of the first block table(16 bit field)

As I mentioned earlier, the ext_tag_desc is 16-bits only and the constant is SPECIAL_LINKED not EXT_LINKED. In addition, the value first_blk_len is not actually stored in the header. It is calculated based on number of blocks, the size of each block, and the total element length.
After I fixed these issues, the numbers started to make more sense. Do you know how I can reach out to the people who write the documentation so that they can solve these problems? It seems that the documentation is outdated.

On Thu, Jun 18, 2015 at 3:04 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com<mailto:aseldawy+hdf@gmail.com>> wrote:
Hi all,

To give some background context, I'm working on a Java project that deals with HDF files. I used to use the Java-interface to the native HDF library, however, I had some issues with it and I wanted to use a pure Java library. I didn't find any pure Java HDF libraries, so I decided to make my own. It's going very fine and it fits very well in my project. It's not complete yet, but I'm extending it bit-by-bit. All my work is open source. Currently, it's part of a bigger SpatialHadoop project [https://github.com/aseldawy/spatialhadoop2]. However, I'm planning to ship it is a standalone library which will be available for the community.

Currently, I'm working on the Linked Block feature of HDF files. However, I'm having problems with this file
http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.02.18/MOD13Q1.A2000049.h10v11.005.2006270195941.hdf
The file opens fine in HDF viewer, which means that the file is correct. I'm following the HDF specs at
http://www.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Linked block elements are described in Section 10.3 on page 149 (160 in the PDF). First, there seems to be a problem in the documentation as it says that EXT_LINKED constant is 32-bits. This seems to be error because other extension identifiers are 16-bits (e.g., see SPECIAL_EXT in 10.4).
Anyway, I fixed it myself and worked with a 16-bit identifier. However, I started to see some problems and I don't know if my fix of 32-bits to 16-bits was right or wrong.
The file I mentioned above has an extended data block with tagID 1963 (DFTAG_VS) and reference no 49. The data of this object is located in offset 38215 and length 16 bytes. When I parse this block according to the specs, I get the following numbers which are inconsistent.
Length: 57600
first_len: 4096
blk_len: 16
num_blk: 131072
Now these numbers do not make sense because if the length of the first block is 4096 and then we have 131072 blocks, each of 16-bytes length, the total length would be 2101248 which is much larger than 57600.
I'm having more troubles when parsing the linked block table but let's resolve this issue first as the later might be related.

9

I couldn't find a link to download the tool. Can you please point me to the
page that contains the download?
I've been looking at
http://www.hdfgroup.org/products/hdf4_tools
and
http://www.hdfgroup.org/products/hdf4_tools/toolsbycat.html
but I couldn't find anything related to mapping.

···

On Thu, Jul 2, 2015 at 4:52 AM, Ahmed Eldawy <aseldawy+hdf@gmail.com> wrote:

I've seen your previous comment about the HDF4 Mapping Tool. However, I
couldn't find any instructions on how to install or use it. If the
generated XML file looks like this

http://e4ftl01.cr.usgs.gov/MOLA/MYD11A1.005/2002.07.08/MYD11A1.A2002189.h18v06.005.2007216142720.hdf.xml
then I don't think it will be of much help to me.

On Sun, Jun 28, 2015 at 11:50 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com> > wrote:

Hi Binh-Minh Ribler,
  Thanks for your response. I've been looking into the issue myself and I
think I found the problem. I had to check the source code of the HDF
library and I found some useful comments in the file hblocks.c. According
to this file, the correct header format is
   ext_tag_desc - SPECIAL_LINKED(16 bit constant), identifies this as a
linked block description record
   elem_tot_len - Length of the entire element(32 bit field)
   blk_length - Length of successive data blocks(32 bit field) after
first block,
   first block is calculated.
   num_blk - Number of blocks per block table(32 bit field)
   link_ref - Reference number of the first block table(16 bit
field)

As I mentioned earlier, the ext_tag_desc is 16-bits only and the constant
is SPECIAL_LINKED not EXT_LINKED. In addition, the value first_blk_len is
not actually stored in the header. It is calculated based on number of
blocks, the size of each block, and the total element length.
After I fixed these issues, the numbers started to make more sense. Do
you know how I can reach out to the people who write the documentation so
that they can solve these problems? It seems that the documentation is
outdated.

On Thu, Jun 18, 2015 at 3:04 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com> >> wrote:

Hi all,

To give some background context, I'm working on a Java project that
deals with HDF files. I used to use the Java-interface to the native HDF
library, however, I had some issues with it and I wanted to use a pure Java
library. I didn't find any pure Java HDF libraries, so I decided to make
my own. It's going very fine and it fits very well in my project. It's not
complete yet, but I'm extending it bit-by-bit. All my work is open source.
Currently, it's part of a bigger SpatialHadoop project [
https://github.com/aseldawy/spatialhadoop2]. However, I'm planning to
ship it is a standalone library which will be available for the community.

Currently, I'm working on the Linked Block feature of HDF files.
However, I'm having problems with this file

http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.02.18/MOD13Q1.A2000049.h10v11.005.2006270195941.hdf
The file opens fine in HDF viewer, which means that the file is correct.
I'm following the HDF specs at
http://www.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Linked block elements are described in Section 10.3 on page 149 (160 in
the PDF). First, there seems to be a problem in the documentation as it
says that EXT_LINKED constant is 32-bits. This seems to be error because
other extension identifiers are 16-bits (e.g., see SPECIAL_EXT in 10.4).
Anyway, I fixed it myself and worked with a 16-bit identifier. However,
I started to see some problems and I don't know if my fix of 32-bits to
16-bits was right or wrong.
The file I mentioned above has an extended data block with tagID 1963
(DFTAG_VS) and reference no 49. The data of this object is located in
offset 38215 and length 16 bytes. When I parse this block according to the
specs, I get the following numbers which are inconsistent.
Length: 57600
first_len: 4096
blk_len: 16
num_blk: 131072
Now these numbers do not make sense because if the length of the first
block is 4096 and then we have 131072 blocks, each of 16-bytes length, the
total length would be 2101248 which is much larger than 57600.
I'm having more troubles when parsing the linked block table but let's
resolve this issue first as the later might be related.

10

Hi Ahmed,

Please try to download it from here: https://www.hdfgroup.org/projects/h4map/h4map_writer.html.

However, we are changing from hdfgroup.uiuc.edu to hdfgroup.org so if you have problems with the links, please check back tomorrow.

Thanks,

Binh-Minh

···

________________________________
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Ahmed Eldawy <aseldawy+hdf@gmail.com>
Sent: Monday, July 06, 2015 4:58 PM
To: hdf-forum
Subject: Re: [Hdf-forum] Linked block in HDF4 files

I couldn't find a link to download the tool. Can you please point me to the page that contains the download?
I've been looking at
http://www.hdfgroup.org/products/hdf4_tools
and
http://www.hdfgroup.org/products/hdf4_tools/toolsbycat.html
but I couldn't find anything related to mapping.

On Thu, Jul 2, 2015 at 4:52 AM, Ahmed Eldawy <aseldawy+hdf@gmail.com<mailto:aseldawy+hdf@gmail.com>> wrote:
I've seen your previous comment about the HDF4 Mapping Tool. However, I couldn't find any instructions on how to install or use it. If the generated XML file looks like this
http://e4ftl01.cr.usgs.gov/MOLA/MYD11A1.005/2002.07.08/MYD11A1.A2002189.h18v06.005.2007216142720.hdf.xml
then I don't think it will be of much help to me.

On Sun, Jun 28, 2015 at 11:50 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com<mailto:aseldawy+hdf@gmail.com>> wrote:
Hi Binh-Minh Ribler,
  Thanks for your response. I've been looking into the issue myself and I think I found the problem. I had to check the source code of the HDF library and I found some useful comments in the file hblocks.c. According to this file, the correct header format is
   ext_tag_desc - SPECIAL_LINKED(16 bit constant), identifies this as a linked block description record
   elem_tot_len - Length of the entire element(32 bit field)
   blk_length - Length of successive data blocks(32 bit field) after first block,
   first block is calculated.
   num_blk - Number of blocks per block table(32 bit field)
   link_ref - Reference number of the first block table(16 bit field)

As I mentioned earlier, the ext_tag_desc is 16-bits only and the constant is SPECIAL_LINKED not EXT_LINKED. In addition, the value first_blk_len is not actually stored in the header. It is calculated based on number of blocks, the size of each block, and the total element length.
After I fixed these issues, the numbers started to make more sense. Do you know how I can reach out to the people who write the documentation so that they can solve these problems? It seems that the documentation is outdated.

On Thu, Jun 18, 2015 at 3:04 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com<mailto:aseldawy+hdf@gmail.com>> wrote:
Hi all,

To give some background context, I'm working on a Java project that deals with HDF files. I used to use the Java-interface to the native HDF library, however, I had some issues with it and I wanted to use a pure Java library. I didn't find any pure Java HDF libraries, so I decided to make my own. It's going very fine and it fits very well in my project. It's not complete yet, but I'm extending it bit-by-bit. All my work is open source. Currently, it's part of a bigger SpatialHadoop project [https://github.com/aseldawy/spatialhadoop2]. However, I'm planning to ship it is a standalone library which will be available for the community.

Currently, I'm working on the Linked Block feature of HDF files. However, I'm having problems with this file
http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.02.18/MOD13Q1.A2000049.h10v11.005.2006270195941.hdf
The file opens fine in HDF viewer, which means that the file is correct. I'm following the HDF specs at
http://www.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Linked block elements are described in Section 10.3 on page 149 (160 in the PDF). First, there seems to be a problem in the documentation as it says that EXT_LINKED constant is 32-bits. This seems to be error because other extension identifiers are 16-bits (e.g., see SPECIAL_EXT in 10.4).
Anyway, I fixed it myself and worked with a 16-bit identifier. However, I started to see some problems and I don't know if my fix of 32-bits to 16-bits was right or wrong.
The file I mentioned above has an extended data block with tagID 1963 (DFTAG_VS) and reference no 49. The data of this object is located in offset 38215 and length 16 bytes. When I parse this block according to the specs, I get the following numbers which are inconsistent.
Length: 57600
first_len: 4096
blk_len: 16
num_blk: 131072
Now these numbers do not make sense because if the length of the first block is 4096 and then we have 131072 blocks, each of 16-bytes length, the total length would be 2101248 which is much larger than 57600.
I'm having more troubles when parsing the linked block table but let's resolve this issue first as the later might be related.

11

Thanks again Bih-Minh for your help. I was able to compile and run the hdf4
mapping tool. I think the detailed output will be very helpful for me as I
parse the HDF4 files. I currently have a problem with chunked data but I
know how to handle it from the output of the h4map tool.

···

On Mon, Jul 6, 2015 at 3:58 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com> wrote:

I couldn't find a link to download the tool. Can you please point me to
the page that contains the download?
I've been looking at
http://www.hdfgroup.org/products/hdf4_tools
and
http://www.hdfgroup.org/products/hdf4_tools/toolsbycat.html
but I couldn't find anything related to mapping.

On Thu, Jul 2, 2015 at 4:52 AM, Ahmed Eldawy <aseldawy+hdf@gmail.com> > wrote:

I've seen your previous comment about the HDF4 Mapping Tool. However, I
couldn't find any instructions on how to install or use it. If the
generated XML file looks like this

http://e4ftl01.cr.usgs.gov/MOLA/MYD11A1.005/2002.07.08/MYD11A1.A2002189.h18v06.005.2007216142720.hdf.xml
then I don't think it will be of much help to me.

On Sun, Jun 28, 2015 at 11:50 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com> >> wrote:

Hi Binh-Minh Ribler,
  Thanks for your response. I've been looking into the issue myself and
I think I found the problem. I had to check the source code of the HDF
library and I found some useful comments in the file hblocks.c. According
to this file, the correct header format is
   ext_tag_desc - SPECIAL_LINKED(16 bit constant), identifies this as
a linked block description record
   elem_tot_len - Length of the entire element(32 bit field)
   blk_length - Length of successive data blocks(32 bit field) after
first block,
   first block is calculated.
   num_blk - Number of blocks per block table(32 bit field)
   link_ref - Reference number of the first block table(16 bit
field)

As I mentioned earlier, the ext_tag_desc is 16-bits only and the
constant is SPECIAL_LINKED not EXT_LINKED. In addition, the value
first_blk_len is not actually stored in the header. It is calculated based
on number of blocks, the size of each block, and the total element length.
After I fixed these issues, the numbers started to make more sense. Do
you know how I can reach out to the people who write the documentation so
that they can solve these problems? It seems that the documentation is
outdated.

On Thu, Jun 18, 2015 at 3:04 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com> >>> wrote:

Hi all,

To give some background context, I'm working on a Java project that
deals with HDF files. I used to use the Java-interface to the native HDF
library, however, I had some issues with it and I wanted to use a pure Java
library. I didn't find any pure Java HDF libraries, so I decided to make
my own. It's going very fine and it fits very well in my project. It's not
complete yet, but I'm extending it bit-by-bit. All my work is open source.
Currently, it's part of a bigger SpatialHadoop project [
https://github.com/aseldawy/spatialhadoop2]. However, I'm planning to
ship it is a standalone library which will be available for the community.

Currently, I'm working on the Linked Block feature of HDF files.
However, I'm having problems with this file

http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.02.18/MOD13Q1.A2000049.h10v11.005.2006270195941.hdf
The file opens fine in HDF viewer, which means that the file is
correct. I'm following the HDF specs at
http://www.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Linked block elements are described in Section 10.3 on page 149 (160 in
the PDF). First, there seems to be a problem in the documentation as it
says that EXT_LINKED constant is 32-bits. This seems to be error because
other extension identifiers are 16-bits (e.g., see SPECIAL_EXT in 10.4).
Anyway, I fixed it myself and worked with a 16-bit identifier. However,
I started to see some problems and I don't know if my fix of 32-bits to
16-bits was right or wrong.
The file I mentioned above has an extended data block with tagID 1963
(DFTAG_VS) and reference no 49. The data of this object is located in
offset 38215 and length 16 bytes. When I parse this block according to the
specs, I get the following numbers which are inconsistent.
Length: 57600
first_len: 4096
blk_len: 16
num_blk: 131072
Now these numbers do not make sense because if the length of the first
block is 4096 and then we have 131072 blocks, each of 16-bytes length, the
total length would be 2101248 which is much larger than 57600.
I'm having more troubles when parsing the linked block table but let's
resolve this issue first as the later might be related.

12

That's great, Ahmed! Thank you for letting us know.

Depending on what you need for your project, you might also like to look into HDFView (HDF file browser) and HDF-Java wrappers at:

https://www.hdfgroup.org/products/java/release/download.html

​Binh-Minh

···

________________________________
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Ahmed Eldawy <aseldawy+hdf@gmail.com>
Sent: Tuesday, July 07, 2015 3:36 PM
To: hdf-forum
Subject: Re: [Hdf-forum] Linked block in HDF4 files

Thanks again Bih-Minh for your help. I was able to compile and run the hdf4 mapping tool. I think the detailed output will be very helpful for me as I parse the HDF4 files. I currently have a problem with chunked data but I know how to handle it from the output of the h4map tool.

On Mon, Jul 6, 2015 at 3:58 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com<mailto:aseldawy+hdf@gmail.com>> wrote:
I couldn't find a link to download the tool. Can you please point me to the page that contains the download?
I've been looking at
http://www.hdfgroup.org/products/hdf4_tools
and
http://www.hdfgroup.org/products/hdf4_tools/toolsbycat.html
but I couldn't find anything related to mapping.

On Thu, Jul 2, 2015 at 4:52 AM, Ahmed Eldawy <aseldawy+hdf@gmail.com<mailto:aseldawy+hdf@gmail.com>> wrote:
I've seen your previous comment about the HDF4 Mapping Tool. However, I couldn't find any instructions on how to install or use it. If the generated XML file looks like this
http://e4ftl01.cr.usgs.gov/MOLA/MYD11A1.005/2002.07.08/MYD11A1.A2002189.h18v06.005.2007216142720.hdf.xml
then I don't think it will be of much help to me.

On Sun, Jun 28, 2015 at 11:50 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com<mailto:aseldawy+hdf@gmail.com>> wrote:
Hi Binh-Minh Ribler,
  Thanks for your response. I've been looking into the issue myself and I think I found the problem. I had to check the source code of the HDF library and I found some useful comments in the file hblocks.c. According to this file, the correct header format is
   ext_tag_desc - SPECIAL_LINKED(16 bit constant), identifies this as a linked block description record
   elem_tot_len - Length of the entire element(32 bit field)
   blk_length - Length of successive data blocks(32 bit field) after first block,
   first block is calculated.
   num_blk - Number of blocks per block table(32 bit field)
   link_ref - Reference number of the first block table(16 bit field)

As I mentioned earlier, the ext_tag_desc is 16-bits only and the constant is SPECIAL_LINKED not EXT_LINKED. In addition, the value first_blk_len is not actually stored in the header. It is calculated based on number of blocks, the size of each block, and the total element length.
After I fixed these issues, the numbers started to make more sense. Do you know how I can reach out to the people who write the documentation so that they can solve these problems? It seems that the documentation is outdated.

On Thu, Jun 18, 2015 at 3:04 PM, Ahmed Eldawy <aseldawy+hdf@gmail.com<mailto:aseldawy+hdf@gmail.com>> wrote:
Hi all,

To give some background context, I'm working on a Java project that deals with HDF files. I used to use the Java-interface to the native HDF library, however, I had some issues with it and I wanted to use a pure Java library. I didn't find any pure Java HDF libraries, so I decided to make my own. It's going very fine and it fits very well in my project. It's not complete yet, but I'm extending it bit-by-bit. All my work is open source. Currently, it's part of a bigger SpatialHadoop project [https://github.com/aseldawy/spatialhadoop2]. However, I'm planning to ship it is a standalone library which will be available for the community.

Currently, I'm working on the Linked Block feature of HDF files. However, I'm having problems with this file
http://e4ftl01.cr.usgs.gov/MOLT/MOD13Q1.005/2000.02.18/MOD13Q1.A2000049.h10v11.005.2006270195941.hdf
The file opens fine in HDF viewer, which means that the file is correct. I'm following the HDF specs at
http://www.hdfgroup.org/release4/doc/DSpec_html/DS.pdf
Linked block elements are described in Section 10.3 on page 149 (160 in the PDF). First, there seems to be a problem in the documentation as it says that EXT_LINKED constant is 32-bits. This seems to be error because other extension identifiers are 16-bits (e.g., see SPECIAL_EXT in 10.4).
Anyway, I fixed it myself and worked with a 16-bit identifier. However, I started to see some problems and I don't know if my fix of 32-bits to 16-bits was right or wrong.
The file I mentioned above has an extended data block with tagID 1963 (DFTAG_VS) and reference no 49. The data of this object is located in offset 38215 and length 16 bytes. When I parse this block according to the specs, I get the following numbers which are inconsistent.
Length: 57600
first_len: 4096
blk_len: 16
num_blk: 131072
Now these numbers do not make sense because if the length of the first block is 4096 and then we have 131072 blocks, each of 16-bytes length, the total length would be 2101248 which is much larger than 57600.
I'm having more troubles when parsing the linked block table but let's resolve this issue first as the later might be related.