(To resurrect a variant on a long-dead thread...
http://hdf-forum.184993.n3.nabble.com/H5Fclose-hangs-when-using-parallel-HDF5-td3369047.html)
I have an MPI parallel application that is reproducibly hanging within
H5Fclose for HDF5 1.8.12 on OpenMPI/1.6.5. The application
always collectively writes HDF5 metadata to the best of my knowledge.
The hang occurs after opening, writing to, and closing two files in
quick succession. I'm quite stuck here, and would appreciate any
thoughts for how to debug this.
Out of 96 total ranks, I see two different distinct backtraces at the
time of the hang. On 88 ranks I observe
#6 0x00002b77dcd5f402 in PMPI_Barrier (comm=0x2b7ff40) at pbarrier.c:70
#7 0x00002b77dc279455 in H5FD_mpio_truncate () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#8 0x00002b77dc2729b9 in H5FD_truncate () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#9 0x00002b77dc260559 in H5F_dest () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#10 0x00002b77dc2619bd in H5F_try_close () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#11 0x00002b77dc261cc4 in H5F_close () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#12 0x00002b77dc2d38a2 in H5I_dec_ref () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#13 0x00002b77dc2d3952 in H5I_dec_app_ref () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#14 0x00002b77dc2625da in H5Fclose () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#15 0x00002b77dbd64151 in esio_file_close (h=0x2b79020) at esio.c:730
#16 0x00002b77dbd642eb in esio_file_close_restart (h=0x2b79020,
restart_template=0x2686f58 "sample#####.h5", retain_count=32768) at
esio.c:770
while on the remaining 8 ranks I see
#6 0x00002b517f4e7402 in PMPI_Barrier (comm=0x32d8640) at pbarrier.c:70
#7 0x00002b518ff4868c in mca_io_romio_dist_MPI_File_close
(mpi_fh=0x32faa80) at close.c:62
#8 0x00002b517f4d4b6e in file_destructor (file=0x32d8300) at file/file.c:273
#9 0x00002b517f4d4a89 in opal_obj_run_destructors (object=0x32d8300)
at ../opal/class/opal_object.h:448
#10 ompi_file_close (file=0x32f8c20) at file/file.c:146
#11 0x00002b517f501c48 in PMPI_File_close (fh=0x32f8c20) at pfile_close.c:59
#12 0x00002b517ea02454 in H5FD_mpio_close () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#13 0x00002b517e9f8a61 in H5FD_close () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#14 0x00002b517e9e84fb in H5F_dest () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#15 0x00002b517e9e99bd in H5F_try_close () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#16 0x00002b517e9e9cc4 in H5F_close () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#17 0x00002b517ea5b8a2 in H5I_dec_ref () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#18 0x00002b517ea5b952 in H5I_dec_app_ref () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#19 0x00002b517e9ea5da in H5Fclose () from
/home1/rhys/Stow/hdf5-1.8.12/lib/libhdf5.so.8
#20 0x00002b517e4ec151 in esio_file_close (h=0x32f9100) at esio.c:730
#21 0x00002b517e4ec2eb in esio_file_close_restart (h=0x32f9100,
restart_template=0x262f6d8 "sample#####.h5", retain_count=32768) at
esio.c:770
with everything underneath level #6 being OpenMPI's MPI_Barrier
details. The application from frames #20, #21, and higher (not shown)
are the same.
The H5FD_truncate in frame #8 above call seems to be near
src/H5F.c:1140 within the HDF 1.8.12 source code. My H5pubconf.h does
show H5_MPI_FILE_SET_SIZE_BIG to be defined to 1 which might matter
since that makes it look like the PMPI_Barrier that's hanging is
src/H5FDmpio.c:2020 on 88/96 ranks while on the remaining 8/96 ranks
it's a barrier hidden underneath the MPI_File_close call at
src/H5FDmpic.c:1174.
The final 8/96 ranks are on a node doing a bit more work with
opening/saving the sequence of two files. Specifically, one rank on
that node is responsible for moving the first file after it is closed
and for copying an auxiliary file into position before it is
collectively opened. Those operations use some basic POSIX calls. I
have tried to be careful to flush those changes, and have I
incorporated MPI Barriers to prevent the other ranks from rushing
ahead.
The same code runs without issue on mpich2/1.4.1 with HDF 1.8.9. It
has also run on TACC's Lonestar resource which is, I think, mvapich
and HDF/1.8.5-p1. I can't check the mvapich version at the moment as
Lonestar is down tonight.
Thanks for your time,
Rhys