Problem writing 3GB to a data set.

Hdf-forum archives
1

I am trying to write out a single array of size 3,101,590,800 bytes and I am getting the following HDF error.

Writing Cell Data 'EulerAngles' to HDF5 File
HDF5-DIAG: Error detected in HDF5 (1.8.9) thread 0:
  #000: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dio.c line 266 in H5Dwrite(): can't write data
    major: Dataset
    minor: Write failed
  #001: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dio.c line 673 in H5D_write(): can't write data
    major: Dataset
    minor: Write failed
  #002: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dcontig.c line 597 in H5D_contig_write(): contiguous write failed
    major: Dataset
    minor: Write failed
  #003: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dselect.c line 306 in H5D_select_write(): write error
    major: Dataspace
    minor: Write failed
  #004: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dselect.c line 217 in H5D_select_io(): write error
    major: Dataspace
    minor: Write failed
  #005: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dcontig.c line 1210 in H5D_contig_writevv(): can't perform vectorized sieve buffer write
    major: Dataset
    minor: Can't operate on object
  #006: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5V.c line 1457 in H5V_opvv(): can't perform operation
    major: Internal error (too specific to document in detail)
    minor: Can't operate on object
  #007: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dcontig.c line 962 in H5D_contig_writevv_sieve_cb(): block write failed
    major: Dataset
    minor: Write failed
  #008: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Fio.c line 158 in H5F_block_write(): write through metadata accumulator failed
    major: Low-level I/O
    minor: Write failed
  #009: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Faccum.c line 808 in H5F_accum_write(): file write failed
    major: Low-level I/O
    minor: Write failed
  #010: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5FDint.c line 185 in H5FD_write(): driver write request failed
    major: Virtual File Layer
    minor: Write failed
  #011: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5FDsec2.c line 852 in H5FD_sec2_write(): file write failed: time = Mon Oct 29 15:34:22 2012
, filename = '/tmp/out.dream3d', file descriptor = 14, errno = 22, error message = 'Invalid argument', buf = 0x26dc5a000, size = 3101590800, offset = 6624
    major: Low-level I/O
    minor: Write failed

Here are some details about the data array
Error Writing Data 'EulerAngles'
    rank=2
    dim[0]=258465900
    dim[1]=3
    TotalElements=775397700
    Size of Type (Bytes):4
    TotalBytes to Write: 3101590800

THis is with HDF version 1.8.9 compiled for 64 bit on OS X 10.6.8. Is there something special I need to be doing in order to write more than 2^32? bytes of data? Or maybe (most likely) there is a bug in how I am calling the HDF5 functions to write this data?

Thanks.

···

___________________________________________________________
Mike Jackson Principal Software Engineer
BlueQuartz Software Dayton, Ohio
mike.jackson@bluequartz.net www.bluequartz.net

2

Hi Mike,

THis is with HDF version 1.8.9 compiled for 64 bit on OS X 10.6.8. Is there something special I need to be doing in order to write more than 2^32? bytes of data? Or maybe (most likely) there is a bug in how I am calling the HDF5 functions to write this data?

The VFDs break down reads and writes that are too large into several
smaller writes (too large = larger than the underlying function calls
can handle. In particular the return type, which is signed, unlike
the "# of bytes to read/write" parameter, which is not). For all
systems other than Windows, this is SSIZET_MAX. So the driver should
handle very large writes just fine, even on 32-bit filesystems.

I'll take a look at this tomorrow.

Dana

3

I have been trying to track this down and have verified that if I try to write data that is 2^31 + 1 bytes then I get the error. If I try to write 2^31 then I am fine and the write succeeds. I am going to try this same code on my Visual Studio 2008 build and see what happens.

···

On Oct 29, 2012, at 3:42 PM, Michael Jackson wrote:

I am trying to write out a single array of size 3,101,590,800 bytes and I am getting the following HDF error.

Writing Cell Data 'EulerAngles' to HDF5 File
HDF5-DIAG: Error detected in HDF5 (1.8.9) thread 0:
#000: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dio.c line 266 in H5Dwrite(): can't write data
   major: Dataset
   minor: Write failed
#001: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dio.c line 673 in H5D_write(): can't write data
   major: Dataset
   minor: Write failed
#002: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dcontig.c line 597 in H5D_contig_write(): contiguous write failed
   major: Dataset
   minor: Write failed
#003: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dselect.c line 306 in H5D_select_write(): write error
   major: Dataspace
   minor: Write failed
#004: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dselect.c line 217 in H5D_select_io(): write error
   major: Dataspace
   minor: Write failed
#005: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dcontig.c line 1210 in H5D_contig_writevv(): can't perform vectorized sieve buffer write
   major: Dataset
   minor: Can't operate on object
#006: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5V.c line 1457 in H5V_opvv(): can't perform operation
   major: Internal error (too specific to document in detail)
   minor: Can't operate on object
#007: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Dcontig.c line 962 in H5D_contig_writevv_sieve_cb(): block write failed
   major: Dataset
   minor: Write failed
#008: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Fio.c line 158 in H5F_block_write(): write through metadata accumulator failed
   major: Low-level I/O
   minor: Write failed
#009: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5Faccum.c line 808 in H5F_accum_write(): file write failed
   major: Low-level I/O
   minor: Write failed
#010: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5FDint.c line 185 in H5FD_write(): driver write request failed
   major: Virtual File Layer
   minor: Write failed
#011: /Users/mjackson/Workspace/hdf5-1.8.9/src/H5FDsec2.c line 852 in H5FD_sec2_write(): file write failed: time = Mon Oct 29 15:34:22 2012
, filename = '/tmp/out.dream3d', file descriptor = 14, errno = 22, error message = 'Invalid argument', buf = 0x26dc5a000, size = 3101590800, offset = 6624
   major: Low-level I/O
   minor: Write failed

Here are some details about the data array
Error Writing Data 'EulerAngles'
   rank=2
   dim[0]=258465900
   dim[1]=3
   TotalElements=775397700
   Size of Type (Bytes):4
   TotalBytes to Write: 3101590800

THis is with HDF version 1.8.9 compiled for 64 bit on OS X 10.6.8. Is there something special I need to be doing in order to write more than 2^32? bytes of data? Or maybe (most likely) there is a bug in how I am calling the HDF5 functions to write this data?

Thanks.

---
Mike Jackson