Problem with MPIPOSIX driver

Boyan_Bejanov · January 3, 2014, 12:11am

Hello,

I am trying to build parallel hdf5 1.8.12 on RHEL6.4 using the Intel compilers (C icc and Fortran ifort) and the IntelMPI 4.1.0 (which are included in the Intel Cluster Studio XE 2013). Everything configured and compiled okay, although I am getting failures when running "make check", specifically in "testpar/testphdf5". The error message is the following:

...
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
...

I have managed to narrow it down - this error is generated when the MPIPOSIX driver is used (I think it's in the call to H5Pset_fapl_mpiposix() itself). The "nocolcause" test is in the function "no_collective_cause_tests" in file "testpar/t_dest.c". When I comment out the two lines in this function that set the "TEST_SET_MPIPOSIX" flag, thus skipping the MPIPOSIX test, everything else checks successfully.

I am very new to HDF5, and to parallel io at all, so any help with this issue will be much appreciated.
Thanks,
Boyan

PS: While poking about it, I found an inconsequential bug in testpar/t_dest.c on line 3681 (in the same function): the line is
MPI_Comm_size(MPI_COMM_WORLD, &mpi_rank);
while it should be
MPI_Comm_rank(MPI_COMM_WORLD, &mpi_rank);

···

====================================================================================

La version française suit le texte anglais.

------------------------------------------------------------------------------------

This email may contain privileged and/or confidential information, and the Bank of
Canada does not waive any related rights. Any distribution, use, or copying of this
email or the information it contains by other than the intended recipient is
unauthorized. If you received this email in error please delete it immediately from
your system and notify the sender promptly by email that you have done so.

------------------------------------------------------------------------------------

Le présent courriel peut contenir de l'information privilégiée ou confidentielle.
La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute diffusion,
utilisation ou copie de ce courriel ou des renseignements qu'il contient par une
personne autre que le ou les destinataires désignés est interdite. Si vous recevez
ce courriel par erreur, veuillez le supprimer immédiatement et envoyer sans délai à
l'expéditeur un message électronique pour l'aviser que vous avez éliminé de votre
ordinateur toute copie du courriel reçu.

Mohamad_Chaarawi · January 3, 2014, 4:50pm

Hi Boyan,

Thank you for reporting this. I’ll definitely look into it, but wanted to get a quick reply before doing so.

You should not use the MPIPOSIX driver (unless there is a very compelling reason to do so) for parallel I/O and should be using the MPIO driver as you would get much better performance. I think you are just reporting the make check failure you were getting here, but just wanted to make sure that you use the better option.

Did you see this issue only when using intel compilers? i.e. have you tried other compilers and saw the same error?

You are correct about the bug in t_dset.c with comm_size needing to be comm_rank. Fortunately neither the size and rank are used in this function, so I will just take them out for the next release.

Thanks,
Mohamad

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Boyan Bejanov
Sent: Thursday, January 02, 2014 6:11 PM
To: hdf-forum@lists.hdfgroup.org
Cc: Boyan Bejanov
Subject: [Hdf-forum] Problem with MPIPOSIX driver

Hello,

I am trying to build parallel hdf5 1.8.12 on RHEL6.4 using the Intel compilers (C icc and Fortran ifort) and the IntelMPI 4.1.0 (which are included in the Intel Cluster Studio XE 2013). Everything configured and compiled okay, although I am getting failures when running “make check”, specifically in “testpar/testphdf5”. The error message is the following:

...
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
...

I have managed to narrow it down – this error is generated when the MPIPOSIX driver is used (I think it’s in the call to H5Pset_fapl_mpiposix() itself). The “nocolcause” test is in the function “no_collective_cause_tests” in file “testpar/t_dest.c”. When I comment out the two lines in this function that set the “TEST_SET_MPIPOSIX” flag, thus skipping the MPIPOSIX test, everything else checks successfully.

I am very new to HDF5, and to parallel io at all, so any help with this issue will be much appreciated.
Thanks,
Boyan

PS: While poking about it, I found an inconsequential bug in testpar/t_dest.c on line 3681 (in the same function): the line is
MPI_Comm_size(MPI_COMM_WORLD, &mpi_rank);
while it should be
MPI_Comm_rank(MPI_COMM_WORLD, &mpi_rank);

====================================================================================

La version française suit le texte anglais.

------------------------------------------------------------------------------------

This email may contain privileged and/or confidential information, and the Bank of
Canada does not waive any related rights. Any distribution, use, or copying of this
email or the information it contains by other than the intended recipient is
unauthorized. If you received this email in error please delete it immediately from
your system and notify the sender promptly by email that you have done so.

------------------------------------------------------------------------------------

Le présent courriel peut contenir de l'information privilégiée ou confidentielle.
La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute diffusion,
utilisation ou copie de ce courriel ou des renseignements qu'il contient par une
personne autre que le ou les destinataires désignés est interdite. Si vous recevez
ce courriel par erreur, veuillez le supprimer immédiatement et envoyer sans délai à
l'expéditeur un message électronique pour l'aviser que vous avez éliminé de votre
ordinateur toute copie du courriel reçu.

epourmal · January 3, 2014, 5:34pm

Dear FORUM members,

Does anyone use MPIPOSIX driver? We are thinking about retiring the feature in the nearest future (1.8.13?)

Thank you!

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Jan 3, 2014, at 10:50 AM, Mohamad Chaarawi <chaarawi@hdfgroup.org> wrote:

Hi Boyan,

Thank you for reporting this. I’ll definitely look into it, but wanted to get a quick reply before doing so.

You should not use the MPIPOSIX driver (unless there is a very compelling reason to do so) for parallel I/O and should be using the MPIO driver as you would get much better performance. I think you are just reporting the make check failure you were getting here, but just wanted to make sure that you use the better option.

Did you see this issue only when using intel compilers? i.e. have you tried other compilers and saw the same error?

You are correct about the bug in t_dset.c with comm_size needing to be comm_rank. Fortunately neither the size and rank are used in this function, so I will just take them out for the next release.

Thanks,
Mohamad

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Boyan Bejanov
Sent: Thursday, January 02, 2014 6:11 PM
To: hdf-forum@lists.hdfgroup.org
Cc: Boyan Bejanov
Subject: [Hdf-forum] Problem with MPIPOSIX driver

Hello,

I am trying to build parallel hdf5 1.8.12 on RHEL6.4 using the Intel compilers (C icc and Fortran ifort) and the IntelMPI 4.1.0 (which are included in the Intel Cluster Studio XE 2013). Everything configured and compiled okay, although I am getting failures when running “make check”, specifically in “testpar/testphdf5”. The error message is the following:

...
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Testing -- test cause for broken collective io (nocolcause)
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
Fatal error in PMPI_Barrier: Invalid communicator, error stack:
PMPI_Barrier(949): MPI_Barrier(comm=0x0) failed
PMPI_Barrier(903): Invalid communicator
...

I have managed to narrow it down – this error is generated when the MPIPOSIX driver is used (I think it’s in the call to H5Pset_fapl_mpiposix() itself). The “nocolcause” test is in the function “no_collective_cause_tests” in file “testpar/t_dest.c”. When I comment out the two lines in this function that set the “TEST_SET_MPIPOSIX” flag, thus skipping the MPIPOSIX test, everything else checks successfully.

I am very new to HDF5, and to parallel io at all, so any help with this issue will be much appreciated.
Thanks,
Boyan

PS: While poking about it, I found an inconsequential bug in testpar/t_dest.c on line 3681 (in the same function): the line is
MPI_Comm_size(MPI_COMM_WORLD, &mpi_rank);
while it should be
MPI_Comm_rank(MPI_COMM_WORLD, &mpi_rank);

====================================================================================

La version française suit le texte anglais.

------------------------------------------------------------------------------------

This email may contain privileged and/or confidential information, and the Bank of
Canada does not waive any related rights. Any distribution, use, or copying of this
email or the information it contains by other than the intended recipient is
unauthorized. If you received this email in error please delete it immediately from
your system and notify the sender promptly by email that you have done so.

------------------------------------------------------------------------------------

Le présent courriel peut contenir de l'information privilégiée ou confidentielle.
La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute diffusion,
utilisation ou copie de ce courriel ou des renseignements qu'il contient par une
personne autre que le ou les destinataires désignés est interdite. Si vous recevez
ce courriel par erreur, veuillez le supprimer immédiatement et envoyer sans délai à
l'expéditeur un message électronique pour l'aviser que vous avez éliminé de votre
ordinateur toute copie du courriel reçu.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

Boyan_Bejanov · January 3, 2014, 5:15pm

Hi Mohamad,

Thank you for your reply and for the hint to not use MPIPOSIX.

To answer your questions: I have tried only the Intel MPI and compilers, I have not tried any other compilers and MPI implementations. I am attaching my configure options, in case you need this info. Please let me know if you need any other details from me.

Thanks for your help,
Boyan

my.configure (1.11 KB)

···

====================================================================================

La version française suit le texte anglais.

------------------------------------------------------------------------------------

This email may contain privileged and/or confidential information, and the Bank of
Canada does not waive any related rights. Any distribution, use, or copying of this
email or the information it contains by other than the intended recipient is
unauthorized. If you received this email in error please delete it immediately from
your system and notify the sender promptly by email that you have done so.

------------------------------------------------------------------------------------

Le présent courriel peut contenir de l'information privilégiée ou confidentielle.
La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute diffusion,
utilisation ou copie de ce courriel ou des renseignements qu'il contient par une
personne autre que le ou les destinataires désignés est interdite. Si vous recevez
ce courriel par erreur, veuillez le supprimer immédiatement et envoyer sans délai à
l'expéditeur un message électronique pour l'aviser que vous avez éliminé de votre
ordinateur toute copie du courriel reçu.

robl · January 7, 2014, 3:32pm

Sounds like a good move to me. It only confuses end-users and can't
do much in the way of I/O optimizations.

==rob

···

On Fri, Jan 03, 2014 at 11:34:20AM -0600, Elena Pourmal wrote:

Dear FORUM members,

Does anyone use MPIPOSIX driver? We are thinking about retiring the feature in the nearest future (1.8.13?)

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

Mohamad_Chaarawi · January 6, 2014, 5:07pm

Hi Boyan,

Unfortunately I was not able to replicate your bug.
I tried with icc and ifort but with MPICH 3.0.4 (I do not have access to Intel’s MPI).

Somehow, in your case, the communicator is getting invalidated. I do not think it has anything to do with the MPIPOSIX driver since there is nothing unusual or complicated going on in there with MPI, but then again, I have no clue what the problem is.
If this is the only error reported in make check, then I would not worry about it too much, since the MPIPOSIX driver is very likely to be decommissioned soon.

If you would like to explore this further, please ping me off-list with a debug trace as to where the barrier call fails in the library, and perhaps track where the communicator gets invalidated, since I cannot replicate those myself. This might give me a better clue as to what is going wrong.

Thanks,
Mohamad

···

From: Boyan Bejanov [mailto:bbejanov@bank-banque-canada.ca]
Sent: Friday, January 3, 2014 11:15 AM
To: HDF Users Discussion List
Cc: Mohamad Chaarawi
Subject: RE: Problem with MPIPOSIX driver

Hi Mohamad,

Thank you for your reply and for the hint to not use MPIPOSIX.

To answer your questions: I have tried only the Intel MPI and compilers, I have not tried any other compilers and MPI implementations. I am attaching my configure options, in case you need this info. Please let me know if you need any other details from me.

Thanks for your help,
Boyan

====================================================================================

La version française suit le texte anglais.

------------------------------------------------------------------------------------

This email may contain privileged and/or confidential information, and the Bank of
Canada does not waive any related rights. Any distribution, use, or copying of this
email or the information it contains by other than the intended recipient is
unauthorized. If you received this email in error please delete it immediately from
your system and notify the sender promptly by email that you have done so.

------------------------------------------------------------------------------------

Le présent courriel peut contenir de l'information privilégiée ou confidentielle.
La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute diffusion,
utilisation ou copie de ce courriel ou des renseignements qu'il contient par une
personne autre que le ou les destinataires désignés est interdite. Si vous recevez
ce courriel par erreur, veuillez le supprimer immédiatement et envoyer sans délai à
l'expéditeur un message électronique pour l'aviser que vous avez éliminé de votre
ordinateur toute copie du courriel reçu.

epourmal · January 7, 2014, 7:02pm

Great! Thank you, Rob!

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Jan 7, 2014, at 9:32 AM, Rob Latham <robl@mcs.anl.gov> wrote:

On Fri, Jan 03, 2014 at 11:34:20AM -0600, Elena Pourmal wrote:

Dear FORUM members,

Does anyone use MPIPOSIX driver? We are thinking about retiring the feature in the nearest future (1.8.13?)

Sounds like a good move to me. It only confuses end-users and can't
do much in the way of I/O optimizations.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org