Hi folks,
has anyone gotten good performance with parallel HDF operations
on a lustre filesystem? When our client count grows large we are seeing
really really bad IO rates and sometimes even failures when doing
parallel IO operations.
Thought I would ask the group to see if anyone is seeing good
lustre performance.
thanks.
s
···
--
************************************************************************
Steve Heistand NASA Ames Research Center
SciCon Group Mail Stop 258-6 steve.heistand@nasa.gov (650) 604-4369 Moffett Field, CA 94035-1000
************************************************************************
"Any opinions expressed are those of our alien overloads, not my own."
Looking at some data from sun a while back, it really depends on your MPI-IO layer. For example sun was working on a ROMIO adio driver for lustre that is only in the absolute newest versions. Previous that ROMIO would use the UFS adio driver and performance was poor.
This data was for flash IO in parallel to hdf5.
Talk to your MPI vendor about what integration their MPI-IO system has with lustre.
I personally have been disappointed for how slow it has been with how quickly lustre has be installed on so many systems. I am glad to see the work finally done,
Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp@umich.edu
(734)936-1985
···
On Nov 25, 2008, at 10:22 AM, Steve Heistand wrote:
Hi folks,
has anyone gotten good performance with parallel HDF operations
on a lustre filesystem? When our client count grows large we are seeing
really really bad IO rates and sometimes even failures when doing
parallel IO operations.
Thought I would ask the group to see if anyone is seeing good
lustre performance.
thanks.
s
--
************************************************************************
Steve Heistand NASA Ames Research Center
SciCon Group Mail Stop 258-6 steve.heistand@nasa.gov (650) 604-4369 Moffett Field, CA 94035-1000
************************************************************************
"Any opinions expressed are those of our alien overloads, not my own."
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
Steve - I forwarded your query on to one of our lustre-based users who has seen performance issues but who is not on the forum list. He replied:
> We haven't seen the kinds of failure that Steve is seeing, but you might recommend that he try the MPI-POSIX VFD and make sure the stripe count is an even multiple of the number of processors. As for the reduced performance at high concurrency, that looks like something bad is happening with the lock manager.
As Quincey noted, we are hoping to get some funding next year to work on improving HDF5 performance on lustre. We believe it will be possible to customize/tune HDF5's interaction with the MPI and lustre layers to "serve" data in the manner needed for lustre to perform better. Further changes, such as two-phase or server-directed I/O have also been discussed as ways to improve performance.
The actual work undertaken will depend on the priorities of the client and the funding they have available.
Because the anticipated funding is limited, we are also trying to identify other users whose work is hampered by HDF5 performance on lustre. If there is sufficient need, we may be able to make a strong case to other sources for additional support of this effort.
I encourage those of you who are using HDF5 on lustre to email me directly. I'll send you a questionnaire that will help us prioritize efforts and possibly secure additional funding. << I haven't yet written the questions so it may be a couple of days before you hear back! >>
-Ruth
···
------------------------------------------------------------
Ruth Aydt
Director of Sponsored Projects and Business Development
The HDF Group aydt@hdfgroup.org (217)265-7837
------------------------------------------------------------
On Nov 25, 2008, at 9:22 AM, Steve Heistand wrote:
Hi folks,
has anyone gotten good performance with parallel HDF operations
on a lustre filesystem? When our client count grows large we are seeing
really really bad IO rates and sometimes even failures when doing
parallel IO operations.
Thought I would ask the group to see if anyone is seeing good
lustre performance.
thanks.
s
--
************************************************************************
Steve Heistand NASA Ames Research Center
SciCon Group Mail Stop 258-6 steve.heistand@nasa.gov (650) 604-4369 Moffett Field, CA 94035-1000
************************************************************************
"Any opinions expressed are those of our alien overloads, not my own."
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
Hi all,
Hopefully, we (that is, The HDF Group) will be getting some funding to address this issue, probably in the first half of '09.
Quincey Koziol
···
On Nov 25, 2008, at 10:34 AM, Brock Palen wrote:
Eh not so much here ether.
Looking at some data from sun a while back, it really depends on your MPI-IO layer. For example sun was working on a ROMIO adio driver for lustre that is only in the absolute newest versions. Previous that ROMIO would use the UFS adio driver and performance was poor.
This data was for flash IO in parallel to hdf5.
Talk to your MPI vendor about what integration their MPI-IO system has with lustre.
I personally have been disappointed for how slow it has been with how quickly lustre has be installed on so many systems. I am glad to see the work finally done,
Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp@umich.edu
(734)936-1985
On Nov 25, 2008, at 10:22 AM, Steve Heistand wrote:
Hi folks,
has anyone gotten good performance with parallel HDF operations
on a lustre filesystem? When our client count grows large we are seeing
really really bad IO rates and sometimes even failures when doing
parallel IO operations.
Thought I would ask the group to see if anyone is seeing good
lustre performance.
thanks.
s
--
************************************************************************
Steve Heistand NASA Ames Research Center
SciCon Group Mail Stop 258-6 steve.heistand@nasa.gov (650) 604-4369 Moffett Field, CA 94035-1000
************************************************************************
"Any opinions expressed are those of our alien overloads, not my own."
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
It will be very helpful if this community can share with us (The HDF Group) "simple" HDF5 C and/or Fortran examples that represent I/O patterns of applications with poor performance.
Thanks in advance,
Elena
···
On Nov 25, 2008, at 12:42 PM, Quincey Koziol wrote:
Hi all,
Hopefully, we (that is, The HDF Group) will be getting some funding to address this issue, probably in the first half of '09.
Quincey Koziol
On Nov 25, 2008, at 10:34 AM, Brock Palen wrote:
Eh not so much here ether.
Looking at some data from sun a while back, it really depends on your MPI-IO layer. For example sun was working on a ROMIO adio driver for lustre that is only in the absolute newest versions. Previous that ROMIO would use the UFS adio driver and performance was poor.
This data was for flash IO in parallel to hdf5.
Talk to your MPI vendor about what integration their MPI-IO system has with lustre.
I personally have been disappointed for how slow it has been with how quickly lustre has be installed on so many systems. I am glad to see the work finally done,
Brock Palen
www.umich.edu/~brockp
Center for Advanced Computing
brockp@umich.edu
(734)936-1985
On Nov 25, 2008, at 10:22 AM, Steve Heistand wrote:
Hi folks,
has anyone gotten good performance with parallel HDF operations
on a lustre filesystem? When our client count grows large we are seeing
really really bad IO rates and sometimes even failures when doing
parallel IO operations.
Thought I would ask the group to see if anyone is seeing good
lustre performance.
thanks.
s
--
************************************************************************
Steve Heistand NASA Ames Research Center
SciCon Group Mail Stop 258-6 steve.heistand@nasa.gov (650) 604-4369 Moffett Field, CA 94035-1000
************************************************************************
"Any opinions expressed are those of our alien overloads, not my own."
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
