2gb limit ROMIO FAQ and mpich 3.1 fix?

Hdf-forum archives
1

Hi,

Regarding this 2GB limitation in ROMIO/pHDF5 for which there is an FAQ.

http://www.hdfgroup.org/hdf5-quest.html#p2gb

In mpich 3.1 There where some patches that seem to hint this Romio limitation might be fixed.
Any comets or updates on this and HDF5

Thanks
Rene

2

Hi Rene,

Yes MPICH 3.1 does fix this outstanding issue. HDF5 however has not been updated to take advantage of this bug fix.
We are aware of this, and do have a branch that gets rid of this limitation, but it didn't get enough attention/testing to make it into the 1.8.13 release.

Maybe 1.8.14 if all goes well, but I can't promise.
Thanks for pointing out the FAQ that needs updating now.

Thanks,
Mohamad

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Salmon, Rene
Sent: Thursday, June 05, 2014 2:11 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] 2gb limit ROMIO FAQ and mpich 3.1 fix?

Hi,

Regarding this 2GB limitation in ROMIO/pHDF5 for which there is an FAQ.

http://www.hdfgroup.org/hdf5-quest.html#p2gb

In mpich 3.1 There where some patches that seem to hint this Romio limitation might be fixed.
Any comets or updates on this and HDF5

Thanks
Rene

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

3

Thanks for the update Mohamad. I will keep my fingers crossed for 1.8.14.

Rene

···

On Jun 5, 2014, at 2:25 PM, Mohamad Chaarawi <chaarawi@hdfgroup.org> wrote:

Hi Rene,

Yes MPICH 3.1 does fix this outstanding issue. HDF5 however has not been updated to take advantage of this bug fix.
We are aware of this, and do have a branch that gets rid of this limitation, but it didn't get enough attention/testing to make it into the 1.8.13 release.

Maybe 1.8.14 if all goes well, but I can't promise.
Thanks for pointing out the FAQ that needs updating now.

Thanks,
Mohamad

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Salmon, Rene
Sent: Thursday, June 05, 2014 2:11 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] 2gb limit ROMIO FAQ and mpich 3.1 fix?

Hi,

Regarding this 2GB limitation in ROMIO/pHDF5 for which there is an FAQ.

http://www.hdfgroup.org/hdf5-quest.html#p2gb

In mpich 3.1 There where some patches that seem to hint this Romio limitation might be fixed.
Any comets or updates on this and HDF5

Thanks
Rene

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

4

Hi Rene,

Yes MPICH 3.1 does fix this outstanding issue. HDF5 however has not been updated to take advantage of this bug fix.
We are aware of this, and do have a branch that gets rid of this limitation, but it didn't get enough attention/testing to make it into the 1.8.13 release.

I did the work to make ROMIO and MPICH "large datatype" compatible. Mohamad's branch was one of the good test cases to ensure it was in good shape, but the code could still benefit from some more testing.

if you've got an HDF5 or MPI-IO code that generates large I/O requests, I'd be happy to test it out, or if you test it out let me know if there are any problems.

==rob

···

On 06/05/2014 02:25 PM, Mohamad Chaarawi wrote:

Maybe 1.8.14 if all goes well, but I can't promise.
Thanks for pointing out the FAQ that needs updating now.

Thanks,
Mohamad

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Salmon, Rene
Sent: Thursday, June 05, 2014 2:11 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] 2gb limit ROMIO FAQ and mpich 3.1 fix?

Hi,

Regarding this 2GB limitation in ROMIO/pHDF5 for which there is an FAQ.

http://www.hdfgroup.org/hdf5-quest.html#p2gb

In mpich 3.1 There where some patches that seem to hint this Romio limitation might be fixed.
Any comets or updates on this and HDF5

Thanks
Rene

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

5

Hi,

We did test with mpich-3.1.1 and hdf5-1.8.13 and found that that did not work. Probably because we need hdf5-1.8.14 or whatever branch Mohamad has that is aware of the update/fixes to ROMIO or MPICH.

If you guys can point us to some hdf5 snapshot or branch that is aware of the ROMIO updates we can certainly help test that out with our code.

Thanks
Rene

···

On Jun 12, 2014, at 11:19 AM, Rob Latham <robl@mcs.anl.gov> wrote:

On 06/05/2014 02:25 PM, Mohamad Chaarawi wrote:

Hi Rene,

Yes MPICH 3.1 does fix this outstanding issue. HDF5 however has not been updated to take advantage of this bug fix.
We are aware of this, and do have a branch that gets rid of this limitation, but it didn't get enough attention/testing to make it into the 1.8.13 release.

I did the work to make ROMIO and MPICH "large datatype" compatible. Mohamad's branch was one of the good test cases to ensure it was in good shape, but the code could still benefit from some more testing.

if you've got an HDF5 or MPI-IO code that generates large I/O requests, I'd be happy to test it out, or if you test it out let me know if there are any problems.

==rob

Maybe 1.8.14 if all goes well, but I can't promise.
Thanks for pointing out the FAQ that needs updating now.

Thanks,
Mohamad

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Salmon, Rene
Sent: Thursday, June 05, 2014 2:11 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] 2gb limit ROMIO FAQ and mpich 3.1 fix?

Hi,

Regarding this 2GB limitation in ROMIO/pHDF5 for which there is an FAQ.

http://www.hdfgroup.org/hdf5-quest.html#p2gb

In mpich 3.1 There where some patches that seem to hint this Romio limitation might be fixed.
Any comets or updates on this and HDF5

Thanks
Rene

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5