# Problems reading HDF5 files with IntelMPI? **URL:** \ **Category:** Hdf-forum archives\ **Created:** [October 27, 2014, 1:21pm UTC](https://forum.hdfgroup.org/t/problems-reading-hdf5-files-with-intelmpi/3346 "2014-10-27T13:21:25Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![angelv](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/angelv/32/1095_2.png) [@angelv](https://forum.hdfgroup.org/u/angelv)\ **Post date:** [October 27, 2014, 1:21pm UTC](https://forum.hdfgroup.org/t/problems-reading-hdf5-files-with-intelmpi/3346/1 "2014-10-27T13:21:25Z")
Hi, is anyone aware of troubles with PHDF5 and IntelMPI? A test code to reads an HDF5 file in parallel has trouble when scaling if I run it with IntelMPI, but no trouble if I run it, for example, with POE. I'm using Intel compilers 13.0.1, IntelMPI 4.1.3.049, and HDF5 1.8.10 The code just reads a 800x800x800 HDF5 file, and the times I get for reading it are: 128 procs - 0.7262E+01 1024 procs - 0.9815E+01 1280 procs - 0.9930E+01 1600 procs - ??? (it gest stalled…) But the same code (compiled with the above modules), but submitted with IBM's POE instead of IntelMPI has no trouble with 1600 procs (actually no trouble at all with up to 4096 procs) and it reads the file in 0.8963E+01 secs. Any help appreciated, > **···** > > -- > Ángel de Vicente > [http://www.iac.es/galeria/angelv/](http://www.iac.es/galeria/angelv/) > --------------------------------------------------------------------------------------------- > ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protecci�n de Datos, acceda a [http://www.iac.es/disclaimer.php](http://www.iac.es/disclaimer.php) > WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult [http://www.iac.es/disclaimer.php?lang=en](http://www.iac.es/disclaimer.php?lang=en) ---
**Author:** ![robl](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/e9a140/32.png) [@robl](https://forum.hdfgroup.org/u/robl)\ **Post date:** [October 28, 2014, 1:48am UTC](https://forum.hdfgroup.org/t/problems-reading-hdf5-files-with-intelmpi/3346/2 "2014-10-28T01:48:39Z")
> Hi, > > is anyone aware of troubles with PHDF5 and IntelMPI? A test code to > reads an HDF5 file in parallel has trouble when scaling if I run it with > IntelMPI, but no trouble if I run it, for example, with POE. The curie web site says "Global File System" and "Lustre", so I don't know which one you're using. If it's lustre, maybe this will help you: [https://press3.mcs.anl.gov/romio/2014/06/12/romio-and-intel-mpi/](https://press3.mcs.anl.gov/romio/2014/06/12/romio-and-intel-mpi/) ==rob > **···** > > On 10/27/2014 08:21 AM, Angel de Vicente wrote: > > > I'm using Intel compilers 13.0.1, IntelMPI 4.1.3.049, and HDF5 1.8.10 > > > > The code just reads a 800x800x800 HDF5 file, and the times I get for > > reading it are: > > > > 128 procs - 0.7262E+01 > > 1024 procs - 0.9815E+01 > > 1280 procs - 0.9930E+01 > > 1600 procs - ??? (it gest stalled...) > > > > But the same code (compiled with the above modules), but submitted with > > IBM's POE instead of IntelMPI has no trouble with 1600 procs (actually > > no trouble at all with up to 4096 procs) and it reads the file in > > 0.8963E+01 secs. > > > > Any help appreciated, > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@lists.hdfgroup.org > > [http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum\_lists.hdfgroup.org](http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org) > > Twitter: [x.com](https://twitter.com/hdf5) > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA ---
**Author:** ![angelv](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/angelv/32/1095_2.png) [@angelv](https://forum.hdfgroup.org/u/angelv)\ **Post date:** [October 28, 2014, 9:28am UTC](https://forum.hdfgroup.org/t/problems-reading-hdf5-files-with-intelmpi/3346/3 "2014-10-28T09:28:26Z")
Hi Rob, Rob Latham \ writes: > **···** > > > On 10/27/2014 08:21 AM, Angel de Vicente wrote: > > > > > Hi, > > > > > > is anyone aware of troubles with PHDF5 and IntelMPI? A test code to > > > reads an HDF5 file in parallel has trouble when scaling if I run it with > > > IntelMPI, but no trouble if I run it, for example, with POE. > > > > The curie web site says "Global File System" and "Lustre", so I don't know which > > one you're using. > > > > If it's lustre, maybe this will help you: > > > > [ROMIO and Intel-MPI – ROMIO](https://press3.mcs.anl.gov/romio/2014/06/12/romio-and-intel-mpi/) > > thanks, but this issue is not happening in CURIE, but in MareNostrum, > which uses GPFS. > -- > Ángel de Vicente > [http://www.iac.es/galeria/angelv/](http://www.iac.es/galeria/angelv/) > --------------------------------------------------------------------------------------------- > ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protecci�n de Datos, acceda a [Legal notice | Instituto de Astrofísica de Canarias • IAC](http://www.iac.es/disclaimer.php) > WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult [Legal notice | Instituto de Astrofísica de Canarias • IAC](http://www.iac.es/disclaimer.php?lang=en) ---
**Author:** ![robl](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/e9a140/32.png) [@robl](https://forum.hdfgroup.org/u/robl)\ **Post date:** [October 28, 2014, 2:44pm UTC](https://forum.hdfgroup.org/t/problems-reading-hdf5-files-with-intelmpi/3346/4 "2014-10-28T14:44:32Z")
Good to know. While intel mpi does not include any GPFS optimizations, there's really only one optimization that matters for GPFS writes: aligning ROMIO file domains to file system block boundaries. Set the MPI-IO hint "striping\_unit" to the GPFS block size.   Setting MPI-IO hints through HDF5 requires property lists and some other gyrations. Here's a good example, except you would set different hints: [https://wickie.hlrs.de/platforms/index.php/MPI-IO#Adapting\_HDF5.27s\_MPI\_I.2FO\_parameters\_to\_prevent\_locking\_on\_Lustre](https://wickie.hlrs.de/platforms/index.php/MPI-IO#Adapting_HDF5.27s_MPI_I.2FO_parameters_to_prevent_locking_on_Lustre) Determinig the gpfs block size, if you don't know it already, is as simple as 'stat -f' ==rob > **···** > > On 10/28/2014 04:28 AM, Angel de Vicente wrote: > > > Hi Rob, > > > > Rob Latham \ writes: > > > > > On 10/27/2014 08:21 AM, Angel de Vicente wrote: > > > > > > > Hi, > > > > > > > > is anyone aware of troubles with PHDF5 and IntelMPI? A test code to > > > > reads an HDF5 file in parallel has trouble when scaling if I run it with > > > > IntelMPI, but no trouble if I run it, for example, with POE. > > > > > > The curie web site says "Global File System" and "Lustre", so I don't know which > > > one you're using. > > > > > > If it's lustre, maybe this will help you: > > > > > > [ROMIO and Intel-MPI – ROMIO](https://press3.mcs.anl.gov/romio/2014/06/12/romio-and-intel-mpi/) > > > > thanks, but this issue is not happening in CURIE, but in MareNostrum, > > which uses GPFS. > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA ---
**Author:** ![angelv](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/angelv/32/1095_2.png) [@angelv](https://forum.hdfgroup.org/u/angelv)\ **Post date:** [October 30, 2014, 1:57pm UTC](https://forum.hdfgroup.org/t/problems-reading-hdf5-files-with-intelmpi/3346/5 "2014-10-30T13:57:08Z")
Hi, Rob Latham \ writes: > > > > is anyone aware of troubles with PHDF5 and IntelMPI? A test code to > > > > reads an HDF5 file in parallel has trouble when scaling if I run it with > > > > IntelMPI, but no trouble if I run it, for example, with POE. > > thanks, but this issue is not happening in CURIE, but in MareNostrum, > > which uses GPFS. > > Good to know. While intel mpi does not include any GPFS optimizations, there's > really only one optimization that matters for GPFS writes: aligning ROMIO file > domains to file system block boundaries. > > Set the MPI-IO hint "striping\_unit" to the GPFS block size. But this problem is happening when reading a file, not writing it (in any case, I have tried setting the striping\_unit as well, but no difference). So far I have no idea what is going on. ~1500 procs is where the trouble begins, but the number of processors that breaks the program is not fixed. I run it sucessfully with 1515 processors, then it failed with 1480... Any pointers appreciated, > **···** > > -- > Ángel de Vicente > [http://www.iac.es/galeria/angelv/](http://www.iac.es/galeria/angelv/) > --------------------------------------------------------------------------------------------- > ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protecci�n de Datos, acceda a [Legal notice | Instituto de Astrofísica de Canarias • IAC](http://www.iac.es/disclaimer.php) > WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult [Legal notice | Instituto de Astrofísica de Canarias • IAC](http://www.iac.es/disclaimer.php?lang=en) ---
**Author:** ![robl](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/e9a140/32.png) [@robl](https://forum.hdfgroup.org/u/robl)\ **Post date:** [October 30, 2014, 6:12pm UTC](https://forum.hdfgroup.org/t/problems-reading-hdf5-files-with-intelmpi/3346/6 "2014-10-30T18:12:18Z")
> > Set the MPI-IO hint "striping\_unit" to the GPFS block size. > > But this problem is happening when reading a file, not writing it ah, it's right there in the subject. sorry about that. \>(in > any case, I have tried setting the striping\_unit as well, but no > difference). So far I have no idea what is going on. ~1500 procs is > where the trouble begins, but the number of processors that breaks the > program is not fixed. I run it sucessfully with 1515 processors, then it > failed with 1480... I suppose all one can do is get a backtrace from a few processors (by, for example, attaching to a hung process with gdb) and see if you are stuck in communication or if you are stuck in a case where the processes are making very many teeny-tiny read operations (so not stuck, but performing I/O so poorly as to be making imperceptible progress) ==rob > **···** > > On 10/30/2014 08:57 AM, Angel de Vicente wrote: > > > Any pointers appreciated, > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@lists.hdfgroup.org > > [http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum\_lists.hdfgroup.org](http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org) > > Twitter: [x.com](https://twitter.com/hdf5) > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA ---
**Author:** ![angelv](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/angelv/32/1095_2.png) [@angelv](https://forum.hdfgroup.org/u/angelv)\ **Post date:** [November 10, 2014, 12:08pm UTC](https://forum.hdfgroup.org/t/problems-reading-hdf5-files-with-intelmpi/3346/7 "2014-11-10T12:08:05Z")
Hi, Rob Latham \ writes: > > (in > > any case, I have tried setting the striping\_unit as well, but no > > difference). So far I have no idea what is going on. ~1500 procs is > > where the trouble begins, but the number of processors that breaks the > > program is not fixed. I run it sucessfully with 1515 processors, then it > > failed with 1480... > > I suppose all one can do is get a backtrace from a few processors (by, for > example, attaching to a hung process with gdb) and see if you are stuck in > communication or if you are stuck in a case where the processes are making very > many teeny-tiny read operations (so not stuck, but performing I/O so poorly as > to be making imperceptible progress) I will try to attach to some process and see if I can get somewhere, but the issue seems definitely a communication one: I changed the program so that no actual reading is done, just opening the file and closing it, and still gets hung at the h5fopen\_f call, so for some reason the file cannot even get opened when I go beyond ~1500 procs... Thanks, > **···** > > -- > Ángel de Vicente > [http://www.iac.es/galeria/angelv/](http://www.iac.es/galeria/angelv/) > --------------------------------------------------------------------------------------------- > ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protecci�n de Datos, acceda a [Legal notice | Instituto de Astrofísica de Canarias • IAC](http://www.iac.es/disclaimer.php) > WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult [Legal notice | Instituto de Astrofísica de Canarias • IAC](http://www.iac.es/disclaimer.php?lang=en)