Crash when creating a dataset with chunks >= 2^28 doubles

HDF5 Library
1

Hi,

I obtain MPI errors when trying to create a large 1D dataset with a chunked layout. When the chunk size is set to 2^28 or above, I get the errors:

Fatal error in MPI_Type_create_hindexed: Invalid argument, error stack:
MPI_Type_create_hindexed(162): MPI_Type_create_hindexed(count=2, array_of_blocklengths=0x1c2c210, array_of_displacements=0x1c2c230, MPI_BYTE, newtype=0x7ffd845aadbc) failed
MPI_Type_create_hindexed(105): Invalid value for blocklength, must be non-negative but is -1386741760
In: PMI_Abort(67776268, Fatal error in MPI_Type_create_hindexed: Invalid argument, error stack:
MPI_Type_create_hindexed(162): MPI_Type_create_hindexed(count=2, array_of_blocklengths=0x1c2c210, array_of_displacements=0x1c2c230, MPI_BYTE, newtype=0x7ffd845aadbc) failed
MPI_Type_create_hindexed(105): Invalid value for blocklength, must be non-negative but is -1386741760)

I am using intelmpi 18.0.1 and hdf5 1.8.16.

A minimal case is attached. Note that I use a parallel hdf5 library, but it crashes with only 1 MPI rank.
test_chunks.cpp (1.3 KB)

Am I doing something wrong ? Doubles having 8 bytes on my system, the chunk size should correspond to ~2GB, which is supposed to be supported in the HDF5 doc.

Thanks for the help

2

I tested also with openmpi 2.0.2 and hdf5 1.10.1 giving the same type of error. I am unable, for the moment, to test with newer versions.

cheers