Hello,
this is my first post to this forum, related to my first attempt to use HDF5 as the output data format of my programs, so please forgive me for my almost non-existing knowledge of HDF5. I have the following issue:
When writing the contents of a buffer with a size of more than 2GB to a dataset, an error message is raised, and nothing is written to the dataset. A subsequent h5dump on the resulting HDF5 file suggests that the dataset contains the correct number of elements, but all of them are set to zero. In contrast, using exactly the same code with datasets/buffers smaller than 2GB doesn't raise an error.
I have no idea where to search for the origin of my issue, and Google doesn't return anything useful either. A file holding examples of the error messages is attached.
Any help is greatly appreciated.
Best regards, Burkhard.
ham_err.txt (3.41 KB)
Burkhard,
Here are a couple possible starting points on what (MAY?) be
contributing factors or at issue. Apart from the obvious
error-cascade this action triggers, what caught my eye first in the
ham.txt error transcript was the (2) errors below, concerning invalid
arguments, per the 'size' and 'offset'. It may be worth re-checking
the call argument syntax of your high level write call.
The fact that a problem is apparently encountered when hitting a
2Gb threshold is a bit suspicious, as perhaps relating to a numeric
overrun or wrap-around (and subsequent corruption) of a variable whose
data-type is inappropriate for the offset is stored. The other
thought is to re-assess the memory integrity of the buffer or pointer
to buffer (check data type and value in call) itself. Just a few
starting thoughts.
joe
#011: H5FDsec2.c line 806 in H5FD_sec2_write(): file write failed:
time = Fri Sep 23 13:46:56 2011
, filename = 'ham.h5', file descriptor = 3, errno = 22, error message
= 'Invalid argument', buf = 0x156100000, size = 2481873792, offset =
225430636
major: Low-level I/O
minor: Write failed
#011: H5FDsec2.c line 806 in H5FD_sec2_write(): file write failed:
time = Fri Sep 23 13:46:57 2011
, filename = 'ham.h5', file descriptor = 3, errno = 22, error message
= 'Invalid argument', buf = 0x1e9fe7000, size = 9927495168, offset =
2707304428
major: Low-level I/O
minor: Write failed
···
On Tue, Sep 27, 2011 at 10:04 AM, Schmidt, Burkhard <bs@cpfs.mpg.de> wrote:
Hello,
this is my first post to this forum, related to my first attempt to use HDF5 as the output data format of my programs, so please forgive me for my almost non-existing knowledge of HDF5. I have the following issue:
When writing the contents of a buffer with a size of more than 2GB to a dataset, an error message is raised, and nothing is written to the dataset. A subsequent h5dump on the resulting HDF5 file suggests that the dataset contains the correct number of elements, but all of them are set to zero. In contrast, using exactly the same code with datasets/buffers smaller than 2GB doesn't raise an error.
I have no idea where to search for the origin of my issue, and Google doesn't return anything useful either. A file holding examples of the error messages is attached.
Any help is greatly appreciated.
Best regards, Burkhard.
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
------------------------------------------------------
Joseph Glassy
Lead Software Engineer (UM faculty affiliate/contractor)
NASA Measures (Freeze/Thaw),Rm CFC 429
College of Forestry and Conservation
Univ. Montana, Missoula, MT 59812
Lupine Logic Inc.
www.lupinelogic.com
Scientific and Technical Programming
Hi Josef,
thanks for your message. Please excuse the long delay answering it.
I didn't find any error in my code which triggers the issue with the HDF5 library: As far as I can see, data size, type, array lengths, calling HDF5 functions was all correct. The solution to the problem was to write data in chunks, instead of the whole array at once.
Attached is a little example program, a modification of hdf5-1.8.8/hl/examples/ex_lite1.c, which demonstrates the problem: Just increment SIZE by one (or more), recompile, run, and writing the data array will fail. Should I regard this as a bug, or did I miss something essential?
Best regards, Burkhard.
ex_lite1.c.zip (1.62 KB)
