Error in write data


#1

Dear Experts,

I was using HDF5 to write a very large dataset (about 1Tb) and encountered the following error. Note that the same code with the same compilation runs totally fine for a smaller test (~1 Gb data). So I suspect it is a problem with the memory, but the program didn’t report any memory issue of the hdf5_dataspace_memory.
Any suggestions or help is much appreciated.

Error message:
HDF5-DIAG: Error detected in HDF5 (1.10.5) MPI-process 0:
#000: H5Dio.c line 336 in H5Dwrite(): can’t write data
major: Dataset
minor: Write failed
#001: H5Dio.c line 798 in H5D__write(): unable to initialize storage
major: Dataset
minor: Unable to initialize object
#002: H5Dint.c line 2245 in H5D__alloc_storage(): unable to initialize contiguous storage
major: Low-level I/O
minor: Unable to initialize object
#003: H5Dcontig.c line 173 in H5D__contig_alloc(): unable to reserve file space
major: Low-level I/O
minor: No space available for allocation
#004: H5MF.c line 851 in H5MF_alloc(): allocation failed from aggr/vfd
major: Resource unavailable
minor: Can’t allocate space
#005: H5MFaggr.c line 124 in H5MF_aggr_vfd_alloc(): can’t allocate raw data
major: Resource unavailable
minor: Can’t allocate space
#006: H5MFaggr.c line 221 in H5MF__aggr_alloc(): ‘normal’ file space allocation request will overlap into ‘temporary’ file space
major: Resource unavailable
minor: Out of range


Code :

start[0] = pcsum;
start[1] = 0;

count[0] = pc;
count[1] = get_values_per_blockelement(blocknr);
pcsum += pc;

H5Sselect_hyperslab(hdf5_dataspace_in_file, H5S_SELECT_SET,
                    start, NULL, count, NULL);

dims[0] = pc;
dims[1] = get_values_per_blockelement(blocknr);

if ((hdf5_dataspace_memory = H5Screate_simple(rank, dims, NULL) < 0))
   {
    printf("failed to allocate memory for `hdf5 data space memory' (dims[0]: %lld x dims[1] %lld ~ %g MB).\n", dims[0], dims[1],dims[0]* dims[1]*4 / (1024.0 * 1024.0));
    report_memory_usage(&HighMark_run, "RUN");
    endrun(1238);
   }

hdf5_status =
H5Dwrite(hdf5_dataset, hdf5_datatype,
         hdf5_dataspace_memory,
         hdf5_dataspace_in_file, H5P_DEFAULT, CommBuffer);

H5Sclose(hdf5_dataspace_memory);

Weiguang


#2

Hi,

Can you check if you have enough disk space to write data? H5Screate_simple call doesn’t allocate space on the file system.

Thank you!

Elena