Hyperslab collective writing crash with HDF5 1.8.16

VictorSV · July 19, 2016, 9:08am

Hi all,

I'm developing XH5For <https://github.com/victorsndvg/XH5For>, an OO
fortran light-weight XDMF/HDF5 layer, and now I would like to test its
scalability, but I'm lost with an issue that i'm getting.

I'm going to try to explain it as best as I can.

I'm performing some weak scalability tests on Marenostrum III
<http://www.bsc.es/user-support/mn3.php> (GPFS file system) using
Collective Writing with the Contiguous HyperSlab strategy. I'm running
tests with 1, 16, 32, 64, 128, 256, 512, 1024 and 2048 MPI tasks.

All seems to work as expected except for the 2048 MPI tasks test, where I
think I'm getting an MPI interlock (it's still running without doing
anything till the job time exceeds and the job is killed).

After that, i try to reproduce an error with a number of MPI tasks between
1024-2048, and I could get the following error message while launching a
smaller size job, with 1164 MPI tasks:

HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 1009:

  #000: H5Dio.c line 271 in H5Dwrite(): can't prepare for writing data
    major: Dataset
    minor: Write failed
  #001: H5Dio.c line 352 in H5D__pre_write(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5Dio.c line 789 in H5D__write(): can't write data
    major: Dataset
    minor: Write failed
  #003: H5Dmpio.c line 529 in H5D__contig_collective_write(): couldn't
finish shared collective MPI-IO
    major: Low-level I/O
    minor: Write failed

I have used the following libraries/versions during compilation stage:

   - intel/16.0.1
   - impi/5.1.2.150
   - HDF5/1.8.16-mpi

Here you can see how I open the HDF5 file for Collective Writing:

github.com

victorsndvg/XH5For/blob/master/src/lib/hdf5_handler/hdf5_handler.f90#L531




function hdf5_handler_GetMPIEnvironment(this) result(MPIEnvironment)
!-----------------------------------------------------------------
!< Return a pointer to the MPIEnvironment
!----------------------------------------------------------------- 
    class(hdf5_handler_t), intent(IN) :: this                     !< HDF5 handler type
    class(mpi_env_t), pointer         :: MPIEnvironment           !< MPI Environment
!----------------------------------------------------------------- 
    nullify(MPIEnvironment)
    MPIEnvironment => this%MPIEnvironment
end function hdf5_handler_GetMPIEnvironment




function hdf5_handler_GetHDF5Filename(this, Step) result (HDF5FileName)
!-----------------------------------------------------------------
!< Generate HDF5 Filename depending on time step
!----------------------------------------------------------------- 
    class(hdf5_handler_t),  intent(IN)    :: this                 !< XMDF handler
    integer(I4P), optional, intent(IN)    :: Step                 !< Force step number
    character(len=:), allocatable         :: HDF5FileName         !< Name of the current HDF5 file
!-----------------------------------------------------------------

And here, how I write HyperSlabs:

github.com

victorsndvg/XH5For/blob/master/src/lib/hdf5_handler/contiguous_hyperslab/hdf5_contiguous_hyperslab_handler.f90#L102


    class(spatial_grid_descriptor_t), pointer                 :: SpatialGridDescriptor !< Spatial grid descriptor
!----------------------------------------------------------------- 
!< @TODO: face and edge attributes
#ifdef ENABLE_HDF5
    SpatialGridDescriptor => this%GetSpatialGridDescriptor()
    MPIEnvironment        => this%GetMPIEnvironment()
    assert(associated(SpatialGridDescriptor) .and. associated(MPIEnvironment))
    select case(Center)
        case (XDMF_ATTRIBUTE_CENTER_NODE)
            GlobalNumberOfData = int(SpatialGridDescriptor%GetGlobalNumberOfNodes(),HSIZE_T)
            LocalNumberOfData = int(SpatialGridDescriptor%GetNumberOfNodesPerGridID(ID=MPIEnvironment%get_rank()),HSIZE_T)
            DataOffset = int(SpatialGridDescriptor%GetNodeOffsetPerGridID(ID=MPIEnvironment%get_rank()),HSIZE_T)
        case (XDMF_ATTRIBUTE_CENTER_CELL)
            GlobalNumberOfData = int(SpatialGridDescriptor%GetGlobalNumberOfElements(),HSIZE_T)
            LocalNumberOfData = int(SpatialGridDescriptor%GetNumberOfElementsPerGridID(ID=MPIEnvironment%get_rank()),HSIZE_T)
            DataOffset = int(SpatialGridDescriptor%GetElementOffsetPerGridID(ID=MPIEnvironment%get_rank()),HSIZE_T)
        case (XDMF_ATTRIBUTE_CENTER_GRID)
            GlobalNumberOfData = int(MPIEnvironment%get_comm_size(),HSIZE_T)
            LocalNumberOfData = 1_HSIZE_T
            DataOffset = MPIEnvironment%get_rank()
        case Default

Note: ENABLE_MPI, ENABLE_HDF5. ENABLE_PARALLEL_HDF5 definition flags are

enabled

Could anyone give me some ligth about this?

I would greatly appreciate your help!

Thank you in advance,
Víctor.

VictorSV · July 23, 2016, 6:12pm

Hi again,

In order to help to understand what is happening, I've implemented and
example using the HDF5 library.

You can see it in the following link:

github.com

victorsndvg/XH5For/blob/master/src/examples/hdf5_performance_test/ch_unstructured_hexahedron_perf.f90

!-----------------------------------------------------------------
! XH5For (XDMF parallel partitioned mesh I/O on top of HDF5)
! Copyright (c) 2015 Santiago Badia, Alberto F. Martín, 
! Javier Principe and Víctor Sande.
! All rights reserved.
!
! This library is free software; you can redistribute it and/or
! modify it under the terms of the GNU Lesser General Public
! License as published by the Free Software Foundation; either
! version 3.0 of the License, or (at your option) any later version.
!
! This library is distributed in the hope that it will be useful,
! but WITHOUT ANY WARRANTY; without even the implied warranty of
! MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
! Lesser General Public License for more details.
!
! You should have received a copy of the GNU Lesser General Public
! License along with this library.
!-----------------------------------------------------------------
program ch_unstructured_hexahedron

This file has been truncated. show original

I've compiled and launched it reproducing the previous mail and I'm getting
the same behavior/errors with the new code.

I also compiled and linked against HDF5 1.8.17 by myself with the same
results.

I'm not using any tunning hint ... this could be the problem?

Thanks!
Víctor

···

2016-07-19 11:08 GMT+02:00 victor sv <victorsv@gmail.com>:

Hi all,

I'm developing XH5For <https://github.com/victorsndvg/XH5For>, an OO
fortran light-weight XDMF/HDF5 layer, and now I would like to test its
scalability, but I'm lost with an issue that i'm getting.

I'm going to try to explain it as best as I can.

I'm performing some weak scalability tests on Marenostrum III
<http://www.bsc.es/user-support/mn3.php> (GPFS file system) using
Collective Writing with the Contiguous HyperSlab strategy. I'm running
tests with 1, 16, 32, 64, 128, 256, 512, 1024 and 2048 MPI tasks.

All seems to work as expected except for the 2048 MPI tasks test, where I
think I'm getting an MPI interlock (it's still running without doing
anything till the job time exceeds and the job is killed).

After that, i try to reproduce an error with a number of MPI tasks between
1024-2048, and I could get the following error message while launching a
smaller size job, with 1164 MPI tasks:

HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 1009:

  #000: H5Dio.c line 271 in H5Dwrite(): can't prepare for writing data
    major: Dataset
    minor: Write failed
  #001: H5Dio.c line 352 in H5D__pre_write(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5Dio.c line 789 in H5D__write(): can't write data
    major: Dataset
    minor: Write failed
  #003: H5Dmpio.c line 529 in H5D__contig_collective_write(): couldn't
finish shared collective MPI-IO
    major: Low-level I/O
    minor: Write failed

I have used the following libraries/versions during compilation stage:

   - intel/16.0.1
   - impi/5.1.2.150
   - HDF5/1.8.16-mpi

Here you can see how I open the HDF5 file for Collective Writing:

https://github.com/victorsndvg/XH5For/blob/master/src/lib/hdf5_handler/hdf5_handler.f90#L531

And here, how I write HyperSlabs:

https://github.com/victorsndvg/XH5For/blob/master/src/lib/hdf5_handler/contiguous_hyperslab/hdf5_contiguous_hyperslab_handler.f90#L102

Note: ENABLE_MPI, ENABLE_HDF5. ENABLE_PARALLEL_HDF5 definition flags are

enabled

Could anyone give me some ligth about this?

I would greatly appreciate your help!

Thank you in advance,
Víctor.

brtnfld · July 26, 2016, 2:41pm

I suspect that it is an issue with the impi implementation. I have not been able to compile your code on our systems due to various compilation errors (this is using a different compiler as I don’t have access to impi). Have you tried using mpich (or some other mpi implementation) instead to see if that works? Also, have you tried using 1.10.0?

Scot

···

On Jul 23, 2016, at 1:12 PM, victor sv <victorsv@gmail.com<mailto:victorsv@gmail.com>> wrote:

Hi again,

In order to help to understand what is happening, I've implemented and example using the HDF5 library.

You can see it in the following link:

github.com

victorsndvg/XH5For/blob/master/src/examples/hdf5_performance_test/ch_unstructured_hexahedron_perf.f90

!-----------------------------------------------------------------
! XH5For (XDMF parallel partitioned mesh I/O on top of HDF5)
! Copyright (c) 2015 Santiago Badia, Alberto F. Martín, 
! Javier Principe and Víctor Sande.
! All rights reserved.
!
! This library is free software; you can redistribute it and/or
! modify it under the terms of the GNU Lesser General Public
! License as published by the Free Software Foundation; either
! version 3.0 of the License, or (at your option) any later version.
!
! This library is distributed in the hope that it will be useful,
! but WITHOUT ANY WARRANTY; without even the implied warranty of
! MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
! Lesser General Public License for more details.
!
! You should have received a copy of the GNU Lesser General Public
! License along with this library.
!-----------------------------------------------------------------
program ch_unstructured_hexahedron

This file has been truncated. show original

I've compiled and launched it reproducing the previous mail and I'm getting the same behavior/errors with the new code.

I also compiled and linked against HDF5 1.8.17 by myself with the same results.

I'm not using any tunning hint ... this could be the problem?

Thanks!
Víctor

2016-07-19 11:08 GMT+02:00 victor sv <victorsv@gmail.com<mailto:victorsv@gmail.com>>:
Hi all,

I'm developing XH5For<https://github.com/victorsndvg/XH5For>, an OO fortran light-weight XDMF/HDF5 layer, and now I would like to test its scalability, but I'm lost with an issue that i'm getting.

I'm going to try to explain it as best as I can.

I'm performing some weak scalability tests on Marenostrum III<http://www.bsc.es/user-support/mn3.php> (GPFS file system) using Collective Writing with the Contiguous HyperSlab strategy. I'm running tests with 1, 16, 32, 64, 128, 256, 512, 1024 and 2048 MPI tasks.

All seems to work as expected except for the 2048 MPI tasks test, where I think I'm getting an MPI interlock (it's still running without doing anything till the job time exceeds and the job is killed).

After that, i try to reproduce an error with a number of MPI tasks between 1024-2048, and I could get the following error message while launching a smaller size job, with 1164 MPI tasks:

HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 1009:
  #000: H5Dio.c line 271 in H5Dwrite(): can't prepare for writing data
    major: Dataset
    minor: Write failed
  #001: H5Dio.c line 352 in H5D__pre_write(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5Dio.c line 789 in H5D__write(): can't write data
    major: Dataset
    minor: Write failed
  #003: H5Dmpio.c line 529 in H5D__contig_collective_write(): couldn't finish shared collective MPI-IO
    major: Low-level I/O
    minor: Write failed

I have used the following libraries/versions during compilation stage:

  * intel/16.0.1
  * impi/5.1.2.150<http://5.1.2.150/>
  * HDF5/1.8.16-mpi

Here you can see how I open the HDF5 file for Collective Writing:

github.com

victorsndvg/XH5For/blob/master/src/lib/hdf5_handler/hdf5_handler.f90#L531




function hdf5_handler_GetMPIEnvironment(this) result(MPIEnvironment)
!-----------------------------------------------------------------
!< Return a pointer to the MPIEnvironment
!----------------------------------------------------------------- 
    class(hdf5_handler_t), intent(IN) :: this                     !< HDF5 handler type
    class(mpi_env_t), pointer         :: MPIEnvironment           !< MPI Environment
!----------------------------------------------------------------- 
    nullify(MPIEnvironment)
    MPIEnvironment => this%MPIEnvironment
end function hdf5_handler_GetMPIEnvironment




function hdf5_handler_GetHDF5Filename(this, Step) result (HDF5FileName)
!-----------------------------------------------------------------
!< Generate HDF5 Filename depending on time step
!----------------------------------------------------------------- 
    class(hdf5_handler_t),  intent(IN)    :: this                 !< XMDF handler
    integer(I4P), optional, intent(IN)    :: Step                 !< Force step number
    character(len=:), allocatable         :: HDF5FileName         !< Name of the current HDF5 file
!-----------------------------------------------------------------

And here, how I write HyperSlabs:

github.com

victorsndvg/XH5For/blob/master/src/lib/hdf5_handler/contiguous_hyperslab/hdf5_contiguous_hyperslab_handler.f90#L102


    class(spatial_grid_descriptor_t), pointer                 :: SpatialGridDescriptor !< Spatial grid descriptor
!----------------------------------------------------------------- 
!< @TODO: face and edge attributes
#ifdef ENABLE_HDF5
    SpatialGridDescriptor => this%GetSpatialGridDescriptor()
    MPIEnvironment        => this%GetMPIEnvironment()
    assert(associated(SpatialGridDescriptor) .and. associated(MPIEnvironment))
    select case(Center)
        case (XDMF_ATTRIBUTE_CENTER_NODE)
            GlobalNumberOfData = int(SpatialGridDescriptor%GetGlobalNumberOfNodes(),HSIZE_T)
            LocalNumberOfData = int(SpatialGridDescriptor%GetNumberOfNodesPerGridID(ID=MPIEnvironment%get_rank()),HSIZE_T)
            DataOffset = int(SpatialGridDescriptor%GetNodeOffsetPerGridID(ID=MPIEnvironment%get_rank()),HSIZE_T)
        case (XDMF_ATTRIBUTE_CENTER_CELL)
            GlobalNumberOfData = int(SpatialGridDescriptor%GetGlobalNumberOfElements(),HSIZE_T)
            LocalNumberOfData = int(SpatialGridDescriptor%GetNumberOfElementsPerGridID(ID=MPIEnvironment%get_rank()),HSIZE_T)
            DataOffset = int(SpatialGridDescriptor%GetElementOffsetPerGridID(ID=MPIEnvironment%get_rank()),HSIZE_T)
        case (XDMF_ATTRIBUTE_CENTER_GRID)
            GlobalNumberOfData = int(MPIEnvironment%get_comm_size(),HSIZE_T)
            LocalNumberOfData = 1_HSIZE_T
            DataOffset = MPIEnvironment%get_rank()
        case Default

Note: ENABLE_MPI, ENABLE_HDF5. ENABLE_PARALLEL_HDF5 definition flags are enabled

Could anyone give me some ligth about this?

I would greatly appreciate your help!

Thank you in advance,
Víctor.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

VictorSV · July 26, 2016, 4:09pm

Hi Scott,

thanks for your response.

I've tried with GNU compilers, OpenMPI and HDF5 (1.8.15, 1.8.16, 1.18.17),
but in a local scope or small cluster, not in Marenostrum.

Ok, I'm going to try to compile all this tools to reproduce this
environment and I will tell you if there are some news.

Can you report the compilation errors you are talking about? I can help you
in this point or fix any issue!

Thank you again for your effort!
Víctor.

···

2016-07-26 16:41 GMT+02:00 Scot Breitenfeld <brtnfld@hdfgroup.org>:

I suspect that it is an issue with the impi implementation. I have not
been able to compile your code on our systems due to various compilation
errors (this is using a different compiler as I don’t have access to impi).
Have you tried using mpich (or some other mpi implementation) instead to
see if that works? Also, have you tried using 1.10.0?

Scot

On Jul 23, 2016, at 1:12 PM, victor sv <victorsv@gmail.com> wrote:

Hi again,

In order to help to understand what is happening, I've implemented and
example using the HDF5 library.

You can see it in the following link:

https://github.com/victorsndvg/XH5For/blob/master/src/examples/hdf5_performance_test/ch_unstructured_hexahedron_perf.f90

I've compiled and launched it reproducing the previous mail and I'm
getting the same behavior/errors with the new code.

I also compiled and linked against HDF5 1.8.17 by myself with the same
results.

I'm not using any tunning hint ... this could be the problem?

Thanks!
Víctor

2016-07-19 11:08 GMT+02:00 victor sv <victorsv@gmail.com>:

Hi all,

I'm developing XH5For <https://github.com/victorsndvg/XH5For>, an OO
fortran light-weight XDMF/HDF5 layer, and now I would like to test its
scalability, but I'm lost with an issue that i'm getting.

I'm going to try to explain it as best as I can.

I'm performing some weak scalability tests on Marenostrum III
<http://www.bsc.es/user-support/mn3.php> (GPFS file system) using
Collective Writing with the Contiguous HyperSlab strategy. I'm running
tests with 1, 16, 32, 64, 128, 256, 512, 1024 and 2048 MPI tasks.

All seems to work as expected except for the 2048 MPI tasks test, where I
think I'm getting an MPI interlock (it's still running without doing
anything till the job time exceeds and the job is killed).

After that, i try to reproduce an error with a number of MPI tasks
between 1024-2048, and I could get the following error message while
launching a smaller size job, with 1164 MPI tasks:

HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 1009:

  #000: H5Dio.c line 271 in H5Dwrite(): can't prepare for writing data
    major: Dataset
    minor: Write failed
  #001: H5Dio.c line 352 in H5D__pre_write(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5Dio.c line 789 in H5D__write(): can't write data
    major: Dataset
    minor: Write failed
  #003: H5Dmpio.c line 529 in H5D__contig_collective_write(): couldn't
finish shared collective MPI-IO
    major: Low-level I/O
    minor: Write failed

I have used the following libraries/versions during compilation stage:

   - intel/16.0.1
   - impi/5.1.2.150
   - HDF5/1.8.16-mpi

Here you can see how I open the HDF5 file for Collective Writing:

https://github.com/victorsndvg/XH5For/blob/master/src/lib/hdf5_handler/hdf5_handler.f90#L531

And here, how I write HyperSlabs:

https://github.com/victorsndvg/XH5For/blob/master/src/lib/hdf5_handler/contiguous_hyperslab/hdf5_contiguous_hyperslab_handler.f90#L102

Note: ENABLE_MPI, ENABLE_HDF5. ENABLE_PARALLEL_HDF5 definition flags are

enabled

Could anyone give me some ligth about this?

I would greatly appreciate your help!

Thank you in advance,
Víctor.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

brtnfld · July 26, 2016, 7:57pm

Using hdf5 1.8.17, I ran your program on a lustre system using Cray’s mpi wrappers for the intel compiler on 3072, 1536, 1200, 2016 processes with no issues (it completed in under 20s). If you have not already done so, you might want to make sure the hdf5 parallel test programs all passed.

FYI, I managed to get it to compile by adding the static flag for the compiler, I had to hard code the option in cmake. Here is what I used to compile it,

cmake \
    -D CMAKE_C_FLAGS:STRING="-g" \
    -D CMAKE_Fortran_FLAGS:STRING="-g -static" \
    -D CMAKE_BUILD_TYPE:STRING="Debug" \
    -D XH5For_ENABLE_HDF5=ON \
    -D XH5For_ENABLE_MPI=ON \
    -D XH5For_ENABLE_TESTS=ON \
    -D XH5For_ENABLE_EXAMPLES=ON \
    -D BUILD_SHARED_LIBS=OFF \
    -D HDF5_INCLUDE_DIRS:PATH=$HDF5_DIR/include \
    -D HDF5_LIBRARY_DIRS:PATH=$HDF5_DIR/lib \
    $XH5For

Scot

···

On Jul 26, 2016, at 11:09 AM, victor sv <victorsv@gmail.com<mailto:victorsv@gmail.com>> wrote:

Hi Scott,

thanks for your response.

I've tried with GNU compilers, OpenMPI and HDF5 (1.8.15, 1.8.16, 1.18.17), but in a local scope or small cluster, not in Marenostrum.

Ok, I'm going to try to compile all this tools to reproduce this environment and I will tell you if there are some news.

Can you report the compilation errors you are talking about? I can help you in this point or fix any issue!

Thank you again for your effort!
Víctor.

2016-07-26 16:41 GMT+02:00 Scot Breitenfeld <brtnfld@hdfgroup.org<mailto:brtnfld@hdfgroup.org>>:
I suspect that it is an issue with the impi implementation. I have not been able to compile your code on our systems due to various compilation errors (this is using a different compiler as I don’t have access to impi). Have you tried using mpich (or some other mpi implementation) instead to see if that works? Also, have you tried using 1.10.0?

Scot

On Jul 23, 2016, at 1:12 PM, victor sv <victorsv@gmail.com<mailto:victorsv@gmail.com>> wrote:

Hi again,

In order to help to understand what is happening, I've implemented and example using the HDF5 library.

You can see it in the following link:

github.com

victorsndvg/XH5For/blob/master/src/examples/hdf5_performance_test/ch_unstructured_hexahedron_perf.f90

!-----------------------------------------------------------------
! XH5For (XDMF parallel partitioned mesh I/O on top of HDF5)
! Copyright (c) 2015 Santiago Badia, Alberto F. Martín, 
! Javier Principe and Víctor Sande.
! All rights reserved.
!
! This library is free software; you can redistribute it and/or
! modify it under the terms of the GNU Lesser General Public
! License as published by the Free Software Foundation; either
! version 3.0 of the License, or (at your option) any later version.
!
! This library is distributed in the hope that it will be useful,
! but WITHOUT ANY WARRANTY; without even the implied warranty of
! MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
! Lesser General Public License for more details.
!
! You should have received a copy of the GNU Lesser General Public
! License along with this library.
!-----------------------------------------------------------------
program ch_unstructured_hexahedron

This file has been truncated. show original

I've compiled and launched it reproducing the previous mail and I'm getting the same behavior/errors with the new code.

I also compiled and linked against HDF5 1.8.17 by myself with the same results.

I'm not using any tunning hint ... this could be the problem?

Thanks!
Víctor

2016-07-19 11:08 GMT+02:00 victor sv <victorsv@gmail.com<mailto:victorsv@gmail.com>>:
Hi all,

I'm developing XH5For<https://github.com/victorsndvg/XH5For>, an OO fortran light-weight XDMF/HDF5 layer, and now I would like to test its scalability, but I'm lost with an issue that i'm getting.

I'm going to try to explain it as best as I can.

I'm performing some weak scalability tests on Marenostrum III<http://www.bsc.es/user-support/mn3.php> (GPFS file system) using Collective Writing with the Contiguous HyperSlab strategy. I'm running tests with 1, 16, 32, 64, 128, 256, 512, 1024 and 2048 MPI tasks.

All seems to work as expected except for the 2048 MPI tasks test, where I think I'm getting an MPI interlock (it's still running without doing anything till the job time exceeds and the job is killed).

After that, i try to reproduce an error with a number of MPI tasks between 1024-2048, and I could get the following error message while launching a smaller size job, with 1164 MPI tasks:

HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 1009:
  #000: H5Dio.c line 271 in H5Dwrite(): can't prepare for writing data
    major: Dataset
    minor: Write failed
  #001: H5Dio.c line 352 in H5D__pre_write(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5Dio.c line 789 in H5D__write(): can't write data
    major: Dataset
    minor: Write failed
  #003: H5Dmpio.c line 529 in H5D__contig_collective_write(): couldn't finish shared collective MPI-IO
    major: Low-level I/O
    minor: Write failed

I have used the following libraries/versions during compilation stage:

  * intel/16.0.1
  * impi/5.1.2.150<http://5.1.2.150/>
  * HDF5/1.8.16-mpi

Here you can see how I open the HDF5 file for Collective Writing:

github.com

victorsndvg/XH5For/blob/master/src/lib/hdf5_handler/hdf5_handler.f90#L531




function hdf5_handler_GetMPIEnvironment(this) result(MPIEnvironment)
!-----------------------------------------------------------------
!< Return a pointer to the MPIEnvironment
!----------------------------------------------------------------- 
    class(hdf5_handler_t), intent(IN) :: this                     !< HDF5 handler type
    class(mpi_env_t), pointer         :: MPIEnvironment           !< MPI Environment
!----------------------------------------------------------------- 
    nullify(MPIEnvironment)
    MPIEnvironment => this%MPIEnvironment
end function hdf5_handler_GetMPIEnvironment




function hdf5_handler_GetHDF5Filename(this, Step) result (HDF5FileName)
!-----------------------------------------------------------------
!< Generate HDF5 Filename depending on time step
!----------------------------------------------------------------- 
    class(hdf5_handler_t),  intent(IN)    :: this                 !< XMDF handler
    integer(I4P), optional, intent(IN)    :: Step                 !< Force step number
    character(len=:), allocatable         :: HDF5FileName         !< Name of the current HDF5 file
!-----------------------------------------------------------------

And here, how I write HyperSlabs:

github.com

victorsndvg/XH5For/blob/master/src/lib/hdf5_handler/contiguous_hyperslab/hdf5_contiguous_hyperslab_handler.f90#L102


    class(spatial_grid_descriptor_t), pointer                 :: SpatialGridDescriptor !< Spatial grid descriptor
!----------------------------------------------------------------- 
!< @TODO: face and edge attributes
#ifdef ENABLE_HDF5
    SpatialGridDescriptor => this%GetSpatialGridDescriptor()
    MPIEnvironment        => this%GetMPIEnvironment()
    assert(associated(SpatialGridDescriptor) .and. associated(MPIEnvironment))
    select case(Center)
        case (XDMF_ATTRIBUTE_CENTER_NODE)
            GlobalNumberOfData = int(SpatialGridDescriptor%GetGlobalNumberOfNodes(),HSIZE_T)
            LocalNumberOfData = int(SpatialGridDescriptor%GetNumberOfNodesPerGridID(ID=MPIEnvironment%get_rank()),HSIZE_T)
            DataOffset = int(SpatialGridDescriptor%GetNodeOffsetPerGridID(ID=MPIEnvironment%get_rank()),HSIZE_T)
        case (XDMF_ATTRIBUTE_CENTER_CELL)
            GlobalNumberOfData = int(SpatialGridDescriptor%GetGlobalNumberOfElements(),HSIZE_T)
            LocalNumberOfData = int(SpatialGridDescriptor%GetNumberOfElementsPerGridID(ID=MPIEnvironment%get_rank()),HSIZE_T)
            DataOffset = int(SpatialGridDescriptor%GetElementOffsetPerGridID(ID=MPIEnvironment%get_rank()),HSIZE_T)
        case (XDMF_ATTRIBUTE_CENTER_GRID)
            GlobalNumberOfData = int(MPIEnvironment%get_comm_size(),HSIZE_T)
            LocalNumberOfData = 1_HSIZE_T
            DataOffset = MPIEnvironment%get_rank()
        case Default

Note: ENABLE_MPI, ENABLE_HDF5. ENABLE_PARALLEL_HDF5 definition flags are enabled

Could anyone give me some ligth about this?

I would greatly appreciate your help!

Thank you in advance,
Víctor.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

VictorSV · July 26, 2016, 8:15pm

Thank you Scot,

I've checked XH5For with, intel/16.0.1, openmpi/1.8.5 and HDF5/1.8.17 and
it works with 2048 MPI tasks.
I'm launching more tests to check the performance.

Till the moment I'dont know how to handle CMake Fortran flags to suggest my
preferences instead of force ... I will try to fix it.

I'm happy for receiving your support, Thank you so much again!

Best regards,
Víctor.

···

2016-07-26 21:57 GMT+02:00 Scot Breitenfeld <brtnfld@hdfgroup.org>:

Using hdf5 1.8.17, I ran your program on a lustre system using Cray’s mpi
wrappers for the intel compiler on 3072, 1536, 1200, 2016 processes with no
issues (it completed in under 20s). If you have not already done so, you
might want to make sure the hdf5 parallel test programs all passed.

FYI, I managed to get it to compile by adding the static flag for the
compiler, I had to hard code the option in cmake. Here is what I used to
compile it,

cmake \
    -D CMAKE_C_FLAGS:STRING="-g" \
    -D CMAKE_Fortran_FLAGS:STRING="-g -static" \
    -D CMAKE_BUILD_TYPE:STRING="Debug" \
    -D XH5For_ENABLE_HDF5=ON \
    -D XH5For_ENABLE_MPI=ON \
    -D XH5For_ENABLE_TESTS=ON \
    -D XH5For_ENABLE_EXAMPLES=ON \
    -D BUILD_SHARED_LIBS=OFF \
    -D HDF5_INCLUDE_DIRS:PATH=$HDF5_DIR/include \
    -D HDF5_LIBRARY_DIRS:PATH=$HDF5_DIR/lib \
    $XH5For

Scot

On Jul 26, 2016, at 11:09 AM, victor sv <victorsv@gmail.com> wrote:

Hi Scott,

thanks for your response.

I've tried with GNU compilers, OpenMPI and HDF5 (1.8.15, 1.8.16, 1.18.17),
but in a local scope or small cluster, not in Marenostrum.

Ok, I'm going to try to compile all this tools to reproduce this
environment and I will tell you if there are some news.

Can you report the compilation errors you are talking about? I can help
you in this point or fix any issue!

Thank you again for your effort!
Víctor.

2016-07-26 16:41 GMT+02:00 Scot Breitenfeld <brtnfld@hdfgroup.org>:

I suspect that it is an issue with the impi implementation. I have not
been able to compile your code on our systems due to various compilation
errors (this is using a different compiler as I don’t have access to impi).
Have you tried using mpich (or some other mpi implementation) instead to
see if that works? Also, have you tried using 1.10.0?

Scot

On Jul 23, 2016, at 1:12 PM, victor sv <victorsv@gmail.com> wrote:

Hi again,

In order to help to understand what is happening, I've implemented and
example using the HDF5 library.

You can see it in the following link:

https://github.com/victorsndvg/XH5For/blob/master/src/examples/hdf5_performance_test/ch_unstructured_hexahedron_perf.f90

I've compiled and launched it reproducing the previous mail and I'm
getting the same behavior/errors with the new code.

I also compiled and linked against HDF5 1.8.17 by myself with the same
results.

I'm not using any tunning hint ... this could be the problem?

Thanks!
Víctor

2016-07-19 11:08 GMT+02:00 victor sv <victorsv@gmail.com>:

Hi all,

I'm developing XH5For <https://github.com/victorsndvg/XH5For>, an OO
fortran light-weight XDMF/HDF5 layer, and now I would like to test its
scalability, but I'm lost with an issue that i'm getting.

I'm going to try to explain it as best as I can.

I'm performing some weak scalability tests on Marenostrum III
<http://www.bsc.es/user-support/mn3.php> (GPFS file system) using
Collective Writing with the Contiguous HyperSlab strategy. I'm running
tests with 1, 16, 32, 64, 128, 256, 512, 1024 and 2048 MPI tasks.

All seems to work as expected except for the 2048 MPI tasks test, where
I think I'm getting an MPI interlock (it's still running without doing
anything till the job time exceeds and the job is killed).

After that, i try to reproduce an error with a number of MPI tasks
between 1024-2048, and I could get the following error message while
launching a smaller size job, with 1164 MPI tasks:

HDF5-DIAG: Error detected in HDF5 (1.8.16) MPI-process 1009:

  #000: H5Dio.c line 271 in H5Dwrite(): can't prepare for writing data
    major: Dataset
    minor: Write failed
  #001: H5Dio.c line 352 in H5D__pre_write(): can't write data
    major: Dataset
    minor: Write failed
  #002: H5Dio.c line 789 in H5D__write(): can't write data
    major: Dataset
    minor: Write failed
  #003: H5Dmpio.c line 529 in H5D__contig_collective_write(): couldn't
finish shared collective MPI-IO
    major: Low-level I/O
    minor: Write failed

I have used the following libraries/versions during compilation stage:

   - intel/16.0.1
   - impi/5.1.2.150
   - HDF5/1.8.16-mpi

Here you can see how I open the HDF5 file for Collective Writing:

https://github.com/victorsndvg/XH5For/blob/master/src/lib/hdf5_handler/hdf5_handler.f90#L531

And here, how I write HyperSlabs:

https://github.com/victorsndvg/XH5For/blob/master/src/lib/hdf5_handler/contiguous_hyperslab/hdf5_contiguous_hyperslab_handler.f90#L102

Note: ENABLE_MPI, ENABLE_HDF5. ENABLE_PARALLEL_HDF5 definition flags are

enabled

Could anyone give me some ligth about this?

I would greatly appreciate your help!

Thank you in advance,
Víctor.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5