I am using parallel HDF5 version 1.12.1 built on a Linux platform running Ubuntu 20.04.
The too long; didn’t read summary is that I have different number of blocks that I want to write out as a continuous 1-D vector on the solution and I do not see how to write out different number of block per mpi task. My idea of writing out zero length vector doesn’t seem to work.
So the main question is: Is it possible to write different number calls to a dataset? Or is there a better way to do this?
I can provide a simple example code if that helps.
Below is a more detailed discussion:
I am working with something similar to overset grids. In my case I have a solution result that is divided into multiple 5-D arrays on an mpi task. The number of these blocks varies across tasks. There might be one big block on one mpi task and several smaller blocks on a different mpi task.
The blocks can be of different sizes and are NOT continuous in memory. Copy the separate blocks into one continuous memory block is problematic because the memory used by the blocks could be half the total memory available on an mpi task.
What does work is that I am able to write the blocks with a loop over the blocks and treat each block as a one-dimensional vector.
The core loop looks like (yes in Fortran 90):
do iblk = 1, N
starts(1) = blockDataArray(iblk) % qOffset
call H5Screate_simple_f(nDim, gsz, filespace, ierr)
CALL H5Dget_space_f(dset_id, filespace, ierr)
CALL H5Sselect_hyperslab_f (filespace, H5S_SELECT_SET_F, starts, sz, ierr)
print *,"A: myProc: ", myProc, ", iblk:",iblk,", sz: ",sz(1),", gsz: ",gsz(1),", starts: ",starts(1)
call H5Dwrite_f(dset_id, H5T_NATIVE_DOUBLE, blockDataArray(iblk) % qVec, gsz, ierr, &
file_space_id = filespace, mem_space_id = memspace, xfer_prp = plist_id)
call H5Sclose_f(filespace, ierr)
enddo
If the number of blocks (N) is the same on each task this works fine. I can write to the same dataset N times. But if the number of blocks is different, then this hangs. So say processor 0 has 2 blocks and processor 1 has 3 blocks for a two processor case. Then processor is not participating in the above loop. O.K. that makes sense.
So I added a loop that tries to write out zero sized blocks where N is the number of blocks on this task and maxN is the maximum number of blocks on any task.
do iblk = N+1, maxN
starts(1) = 0
sz(1) = 0
call H5Screate_simple_f(nDim, gsz, filespace, ierr)
CALL H5Dget_space_f(dset_id, filespace, ierr)
CALL H5Sselect_hyperslab_f (filespace, H5S_SELECT_SET_F, starts, sz, ierr)
print *,"B: myProc: ", myProc, ", iblk:",iblk,", sz: ",sz(1),", gsz: ",gsz(1),", starts: ",starts(1)
call H5Dwrite_f(dset_id, H5T_NATIVE_DOUBLE, blockDataArray(1) % qVec, gsz, ierr, &
file_space_id = filespace, mem_space_id = memspace, xfer_prp = plist_id)
call H5Sclose_f(filespace, ierr); ASSERT(ierr == 0,"H5Sclose_f")
enddo
This fails with the following error message in trying to call H5Dwrite_f()
HDF5-DIAG: Error detected in HDF5 (1.12.1) MPI-process 0:
#000: H5Dio.c line 291 in H5Dwrite(): can’t write data
major: Dataset
minor: Write failed
#001: H5VLcallback.c line 2113 in H5VL_dataset_write(): dataset write failed
major: Virtual Object Layer
minor: Write failed
#002: H5VLcallback.c line 2080 in H5VL__dataset_write(): dataset write failed
major: Virtual Object Layer
minor: Write failed
#003: H5VLnative_dataset.c line 207 in H5VL__native_dataset_write(): can’t write data
major: Dataset
minor: Write failed
#004: H5Dio.c line 661 in H5D__write(): src and dest dataspaces have different number of elements selected
major: Invalid arguments to routine
minor: Bad value