Efficiently creating and writing to 20,000 datasets

While Kent investigates the file, a note about the h4h5tools binary, it is statically compiled except for the external libraries. I do need to upgrade this binary to include the hdf4, hdf5, szip, zlib, and jpeg headers and static libs to use the h4h5tools library in other projects. Also, I need to update the executables and switch to the static szip and zlib as well.

In addition, I will also verify that the correct external libs are included with hdf4.

Allen

···

Hi Xunlei, Allen and Elena,

I just verified the file on Linux. It also causes problems. So it seems not
only a windows-related bug. We will investigate this and address this soon,
probably within the week.

Kent

On Wed, Jun 30, 2010 at 9:36 AM, Dr. X <xunlei@renci.org> wrote:

> You guys are wonderful!
> Thanks so much.
>
> Best,
> x
>
>
> On 6/30/2010 10:32 AM, Allen D Byrne wrote:
>
>> We will regenerate the hdf4 and h4h5tools binaries to use the same
>> szip/zlib versions as hdf5 1.8.5. We will include the szip/zlib/jpeg dlls
>> with hdf4. In addition we will also include the hdf4 and hdf5 dlls with the
>> h4h5tools binary so that everything will use the same version of libraries.
>> Hopefully this will prevent future problems like this.
>>
>> Allen
>>
>>
>>
>>> Hi Allen and Elena,
>>> Thanks for the help.
>>> I've tried with
>>>
>>> http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/bin/win64-vs2005/hdf5-1.8.4-patch1-win64-vs2005-ivf91-enc.zip
>>>
>>> http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/bin/win64-vs2005/szip-2.1-win64-vs2005-enc.zip
>>>
>>> http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/bin/win64-vs2005/zlib-1.2.3-win64-vs2005.zip
>>>
>>> and
>>>
>>>
>>> http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/bin/win64-vs2008/hdf5-1.8.4-patch1-win64-vs2008-ivf101-enc.zip
>>>
>>> http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/bin/win64-vs2008/szip-2.1-win64-vs2008-enc.zip
>>>
>>> http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/bin/win64-vs2008/zlib-1.2.3-win64-vs2008.zip
>>>
>>> Certainly I did the same trick on copying and renaming szip.* to
>>> szlibdll.*. I got the same error. In my HDF4 distribution, there is no
>>> (external) szip and zlib bundled. Not sure whether szip and zlib are
>>> included in the hdf425.dll...
>>>
>>> Best,
>>> x
>>>
>>> On 6/29/2010 6:04 PM, Elena Pourmal wrote:
>>>
>>>
>>>> Hi Xunlei,
>>>>
>>>> Binaries for h4toh5 were built with the 1.8.4-patch1 release. Could you
>>>> please use the appropriate binary distribution from the
>>>> http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/directory?
>>>>
>>>> Your file is very simple. The conversion tools should work. Please make
>>>> sure you have zlib installed too since some of the datasets use gzip
>>>> compression.
>>>>
>>>> Please let us know if you still have problems.
>>>>
>>>> Thank you!
>>>>
>>>> Elena
>>>> On Jun 29, 2010, at 4:15 PM, Dr. X wrote:
>>>>
>>>>
>>>>
>>>>
>>>>> Hi All,
>>>>> I have a bunch of HDF4 files that I would like to convert to HDF5. So I
>>>>> head to the site to get "H4 / H5 Conversion Library: 2.1.1"
>>>>> http://www.hdfgroup.org/ftp/HDF5/h4toh5/bin/H4H5Tools-2.1.1-win64.zip
>>>>> Then I also downloaded HDF5 and HDF4 from
>>>>>
>>>>> http://www.hdfgroup.org/ftp/HDF5/hdf5-1.8.5/bin/windows/HDF5-1.8.5-win64.zip
>>>>>
>>>>> http://www.hdfgroup.org/ftp/HDF/HDF_Current/bin/windows/HDF4.2.5-win64-vs2005-ivf101.zip
>>>>> After putting the directories into my PATH envrionment and trying to
>>>>> run "h4toh5.exe" from a command line, I got
>>>>> "The program can't start because szlibdll.dll is missing from your
>>>>> computer..." message. So I made a copy of szip.dll and szip.lib and renamed
>>>>> the copy as szlibdll.dll and szlibdll.lib. Then I tried "h4toh5.exe" again.
>>>>> Everything looks fine with usage message printed.
>>>>> However, when I tried with a HDF4 data file as "h4toh5.exe hdf4_file
>>>>> hdf5_file", I got
>>>>> "h4toh5.exe has stopped working A problem caused the program to stop
>>>>> working correctly..." The program crashed.
>>>>>
>>>>> HDFView has no problem viewing that file. And I tested h4toh5.exe with
>>>>> the out.hdf in the ChunkBinary example at
>>>>> http://www.hdfgroup.org/training/hdf4_chunking/ChunkBinary.tar
>>>>> h4toh5.exe worked smoothly. I have uploaded the troubled HDF4 file at
>>>>> ftp://ftp.renci.org/outgoing/dbltrbl100m.hdfgrdbas
>>>>> Would you please take a look? I couldn't tell whether it is the
>>>>> h4toh5's limitation.
>>>>>
>>>>> Thanks a lot.
>>>>>
>>>>> Best,
>>>>> xunlei
>>>>>
>>>>> _______________________________________________
>>>>> Hdf-forum is for HDF software users discussion.
>>>>> Hdf-forum@hdfgroup.org
>>>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Hdf-forum is for HDF software users discussion.
>>>> Hdf-forum@hdfgroup.org
>>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>

The bug inside the conversion tool is found and the fix of the conversion tool should be provided soon.
The problem is due to the long vgroup name in this file. According to the HDF4 reference manual for Vgetname, the maximum length of the vgroup name is defined by VGNAMELENMAX(or 64).
Using vshow to check the example file, you can see
vg:58 <1965/296> (/mnt/ls09/mtkiefer/arps5.2.12/dbltrbl_100m_variable_ss92_fire600//dbltrbl100m.hdfgrdbas {CDF0.0}) has 111 entries:

The vgroup name is "/mnt/ls09/mtkiefer/arps5.2.12/dbltrbl_100m_variable_ss92_fire600//dbltrbl100m.hdfgrdbas". The length is 87, which is greater than 64 characters.
So it causes the "seg fault" for the program.

I use the new HDF4 API Vgetnamelen to obtain the length of vgroup name and dynamically allocate the memory buffer to hold the name. The problem is resolved.

Kent

Allen D Byrne wrote:

···

While Kent investigates the file, a note about the h4h5tools binary, it is statically compiled except for the external libraries. I do need to upgrade this binary to include the hdf4, hdf5, szip, zlib, and jpeg headers and static libs to use the h4h5tools library in other projects. Also, I need to update the executables and switch to the static szip and zlib as well.

In addition, I will also verify that the correct external libs are included with hdf4.

Allen

Hi Xunlei, Allen and Elena,

I just verified the file on Linux. It also causes problems. So it seems not
only a windows-related bug. We will investigate this and address this soon,
probably within the week.

Kent

On Wed, Jun 30, 2010 at 9:36 AM, Dr. X <xunlei@renci.org> wrote:

You guys are wonderful!
Thanks so much.

Best,
x

On 6/30/2010 10:32 AM, Allen D Byrne wrote:

We will regenerate the hdf4 and h4h5tools binaries to use the same
szip/zlib versions as hdf5 1.8.5. We will include the szip/zlib/jpeg dlls
with hdf4. In addition we will also include the hdf4 and hdf5 dlls with the
h4h5tools binary so that everything will use the same version of libraries.
Hopefully this will prevent future problems like this.

Allen

Hi Allen and Elena,
Thanks for the help.
I've tried with

http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/bin/win64-vs2005/hdf5-1.8.4-patch1-win64-vs2005-ivf91-enc.zip

http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/bin/win64-vs2005/szip-2.1-win64-vs2005-enc.zip

http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/bin/win64-vs2005/zlib-1.2.3-win64-vs2005.zip

and

http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/bin/win64-vs2008/hdf5-1.8.4-patch1-win64-vs2008-ivf101-enc.zip

http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/bin/win64-vs2008/szip-2.1-win64-vs2008-enc.zip

http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/bin/win64-vs2008/zlib-1.2.3-win64-vs2008.zip

Certainly I did the same trick on copying and renaming szip.* to
szlibdll.*. I got the same error. In my HDF4 distribution, there is no
(external) szip and zlib bundled. Not sure whether szip and zlib are
included in the hdf425.dll...

Best,
x

On 6/29/2010 6:04 PM, Elena Pourmal wrote:

Hi Xunlei,

Binaries for h4toh5 were built with the 1.8.4-patch1 release. Could you
please use the appropriate binary distribution from the
http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-patch1/directory?

Your file is very simple. The conversion tools should work. Please make
sure you have zlib installed too since some of the datasets use gzip
compression.

Please let us know if you still have problems.

Thank you!

Elena
On Jun 29, 2010, at 4:15 PM, Dr. X wrote:

Hi All,
I have a bunch of HDF4 files that I would like to convert to HDF5. So I
head to the site to get "H4 / H5 Conversion Library: 2.1.1"
http://www.hdfgroup.org/ftp/HDF5/h4toh5/bin/H4H5Tools-2.1.1-win64.zip
Then I also downloaded HDF5 and HDF4 from

http://www.hdfgroup.org/ftp/HDF5/hdf5-1.8.5/bin/windows/HDF5-1.8.5-win64.zip

http://www.hdfgroup.org/ftp/HDF/HDF_Current/bin/windows/HDF4.2.5-win64-vs2005-ivf101.zip
After putting the directories into my PATH envrionment and trying to
run "h4toh5.exe" from a command line, I got
"The program can't start because szlibdll.dll is missing from your
computer..." message. So I made a copy of szip.dll and szip.lib and renamed
the copy as szlibdll.dll and szlibdll.lib. Then I tried "h4toh5.exe" again.
Everything looks fine with usage message printed.
However, when I tried with a HDF4 data file as "h4toh5.exe hdf4_file
hdf5_file", I got
"h4toh5.exe has stopped working A problem caused the program to stop
working correctly..." The program crashed.

HDFView has no problem viewing that file. And I tested h4toh5.exe with
the out.hdf in the ChunkBinary example at
http://www.hdfgroup.org/training/hdf4_chunking/ChunkBinary.tar
h4toh5.exe worked smoothly. I have uploaded the troubled HDF4 file at
ftp://ftp.renci.org/outgoing/dbltrbl100m.hdfgrdbas
Would you please take a look? I couldn't tell whether it is the
h4toh5's limitation.

Thanks a lot.

Best,
xunlei

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
****************************************
Kent Yang
The HDF Group
1901 South First Street, Suite C-2
Champaign, IL 61820

myang6@hdfgroup.org (217)265-5129 (office) (217)333-9049 (fax)
URLs: www.hdfeos.org www.hdfgroup.org
****************************************

Please note that vgroup name and class are no longer restricted to
VGNAMELENMAX(or 64) and, as Kent said, the length can be determined by
Vgetnamelen and Vgetclassnamelen. The HDF4 reference manual will be
updated.

Thanks!
Binh-Minh

···

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-
bounces@hdfgroup.org] On Behalf Of MuQun Yang
Sent: Wednesday, June 30, 2010 7:49 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] failed h4toh5 tool on Windows 64bit

The bug inside the conversion tool is found and the fix of the
conversion tool should be provided soon.
The problem is due to the long vgroup name in this file. According to
the HDF4 reference manual for Vgetname, the maximum length of the vgroup
name is defined by VGNAMELENMAX(or 64).
Using vshow to check the example file, you can see
vg:58 <1965/296>
(/mnt/ls09/mtkiefer/arps5.2.12/dbltrbl_100m_variable_ss92_fire600//dbltrbl1
00m.hdfgrdbas
{CDF0.0}) has 111 entries:

The vgroup name is
"/mnt/ls09/mtkiefer/arps5.2.12/dbltrbl_100m_variable_ss92_fire600//dbltrbl1
00m.hdfgrdbas".
The length is 87, which is greater than 64 characters.
So it causes the "seg fault" for the program.

I use the new HDF4 API Vgetnamelen to obtain the length of vgroup name
and dynamically allocate the memory buffer to hold the name. The problem
is resolved.

Kent

Allen D Byrne wrote:

While Kent investigates the file, a note about the h4h5tools binary, it

is statically compiled except for the external libraries. I do need to
upgrade this binary to include the hdf4, hdf5, szip, zlib, and jpeg headers
and static libs to use the h4h5tools library in other projects. Also, I
need to update the executables and switch to the static szip and zlib as
well.

In addition, I will also verify that the correct external libs are

included with hdf4.

Allen

Hi Xunlei, Allen and Elena,

I just verified the file on Linux. It also causes problems. So it seems

not

only a windows-related bug. We will investigate this and address this

soon,

probably within the week.

Kent

On Wed, Jun 30, 2010 at 9:36 AM, Dr. X <xunlei@renci.org> wrote:

You guys are wonderful!
Thanks so much.

Best,
x

On 6/30/2010 10:32 AM, Allen D Byrne wrote:

We will regenerate the hdf4 and h4h5tools binaries to use the same
szip/zlib versions as hdf5 1.8.5. We will include the szip/zlib/jpeg

dlls

with hdf4. In addition we will also include the hdf4 and hdf5 dlls

with the

h4h5tools binary so that everything will use the same version of

libraries.

Hopefully this will prevent future problems like this.

Allen

Hi Allen and Elena,
Thanks for the help.
I've tried with

http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-

patch1/bin/win64-vs2005/hdf5-1.8.4-patch1-win64-vs2005-ivf91-enc.zip

http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-

patch1/bin/win64-vs2005/szip-2.1-win64-vs2005-enc.zip

http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-

patch1/bin/win64-vs2005/zlib-1.2.3-win64-vs2005.zip

and

http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-

patch1/bin/win64-vs2008/hdf5-1.8.4-patch1-win64-vs2008-ivf101-enc.zip

http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-

patch1/bin/win64-vs2008/szip-2.1-win64-vs2008-enc.zip

http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-

patch1/bin/win64-vs2008/zlib-1.2.3-win64-vs2008.zip

Certainly I did the same trick on copying and renaming szip.* to
szlibdll.*. I got the same error. In my HDF4 distribution, there is

no

(external) szip and zlib bundled. Not sure whether szip and zlib are
included in the hdf425.dll...

Best,
x

On 6/29/2010 6:04 PM, Elena Pourmal wrote:

Hi Xunlei,

Binaries for h4toh5 were built with the 1.8.4-patch1 release. Could

you

please use the appropriate binary distribution from the
http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.4-

patch1/directory?

Your file is very simple. The conversion tools should work. Please

make

sure you have zlib installed too since some of the datasets use gzip
compression.

Please let us know if you still have problems.

Thank you!

Elena
On Jun 29, 2010, at 4:15 PM, Dr. X wrote:

Hi All,
I have a bunch of HDF4 files that I would like to convert to HDF5.

So I

head to the site to get "H4 / H5 Conversion Library: 2.1.1"
http://www.hdfgroup.org/ftp/HDF5/h4toh5/bin/H4H5Tools-2.1.1-

win64.zip

Then I also downloaded HDF5 and HDF4 from

http://www.hdfgroup.org/ftp/HDF5/hdf5-1.8.5/bin/windows/HDF5-1.8.5-

win64.zip

http://www.hdfgroup.org/ftp/HDF/HDF_Current/bin/windows/HDF4.2.5-

win64-vs2005-ivf101.zip

After putting the directories into my PATH envrionment and trying

to

run "h4toh5.exe" from a command line, I got
"The program can't start because szlibdll.dll is missing from your
computer..." message. So I made a copy of szip.dll and szip.lib and

renamed

the copy as szlibdll.dll and szlibdll.lib. Then I tried

"h4toh5.exe" again.

Everything looks fine with usage message printed.
However, when I tried with a HDF4 data file as "h4toh5.exe

hdf4_file

hdf5_file", I got
"h4toh5.exe has stopped working A problem caused the program to

stop

working correctly..." The program crashed.

HDFView has no problem viewing that file. And I tested h4toh5.exe

with

the out.hdf in the ChunkBinary example at
http://www.hdfgroup.org/training/hdf4_chunking/ChunkBinary.tar
h4toh5.exe worked smoothly. I have uploaded the troubled HDF4 file

at

ftp://ftp.renci.org/outgoing/dbltrbl100m.hdfgrdbas
Would you please take a look? I couldn't tell whether it is the
h4toh5's limitation.

Thanks a lot.

Best,
xunlei

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
****************************************
Kent Yang
The HDF Group
1901 South First Street, Suite C-2
Champaign, IL 61820

myang6@hdfgroup.org
(217)265-5129 (office) (217)333-9049 (fax)
URLs:
www.hdfeos.org
www.hdfgroup.org
****************************************

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Internal Virus Database is out-of-date.
Checked by AVG.
Version: 7.5.549 / Virus Database: 270.9.0/1778 - Release Date: 11/9/2008
2:14 PM