Performance hit when writing to big datasets

Hdf-forum archives
1

Hi,

I noticed today something strange. I've been using HDF5 with parallel
processes, and when I raise the dataset size, the library seems to
slow down.
Currently, I have a fixed number of processes writing data from my
simulation. All these processes write 1MB concurrently to contiguous
but exclusive locations in the 1D dataset. When having more than 1
billion elements (for instance), the library is getting really, really
slow (more than 20 times slower than what a usual pwrite would take),
where as nothing can be noticed when writing say 1 million cell.
Is there something fancy going on? I'm trying to access the simplest
part of the library so that the H5write call can be more or less
reduced to a pyrite call, but something is wrong. Should I set
something to tell HF5 than it shouldn't do anything except than
writing data from the calling process to the filesystem (Lustre in my
case, MPI library configured with Lustre support).

Regards,

Matthieu Brucher

···

--
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

2

Hi Matthieu,

No, nothing different happens internally when handling larger datasets.

Some questions to understand more what you are doing:
1) what VFD are you using? I assume it's the MPIO VFD since you said HDF5 with parallel Processes, but you didn't say that explicitly.
2) Did you make sure your Lustre stripe settings (stripe count and size) are appropriate? i.e. the larger amount of data you write, the larger your stripe settings must be.

Those are the only 2 things I can think of without looking at your code.

Thanks,
Mohamad

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Matthieu Brucher
Sent: Tuesday, January 21, 2014 2:25 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] Performance hit when writing to big datasets

Hi,

I noticed today something strange. I've been using HDF5 with parallel processes, and when I raise the dataset size, the library seems to slow down.
Currently, I have a fixed number of processes writing data from my simulation. All these processes write 1MB concurrently to contiguous but exclusive locations in the 1D dataset. When having more than 1 billion elements (for instance), the library is getting really, really slow (more than 20 times slower than what a usual pwrite would take), where as nothing can be noticed when writing say 1 million cell.
Is there something fancy going on? I'm trying to access the simplest part of the library so that the H5write call can be more or less reduced to a pyrite call, but something is wrong. Should I set something to tell HF5 than it shouldn't do anything except than writing data from the calling process to the filesystem (Lustre in my case, MPI library configured with Lustre support).

Regards,

Matthieu Brucher
--
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

3

Hi Mohamad,

Thanks for the reply!
1) I actually don't know, I just turned the flag on
2) I can't change the values on the fly as in the same folder, I have
different kind of files (text, binary, HDF5), and the HDF5 file has to
be created from scratch. Due to the issues with the Lustre API
(license + unstable ABI), I can't integrate it inside the application
:confused:
The current stripe size/count is set for applications that write
bigger datasets/files, so I think they are adequately set up by
default. Anyway, a raw pwrite performs better, so I can tune this
later.

Regards,

Matthieu

···

2014/1/22 Mohamad Chaarawi <chaarawi@hdfgroup.org>:

Hi Matthieu,

No, nothing different happens internally when handling larger datasets.

Some questions to understand more what you are doing:
1) what VFD are you using? I assume it's the MPIO VFD since you said HDF5 with parallel Processes, but you didn't say that explicitly.
2) Did you make sure your Lustre stripe settings (stripe count and size) are appropriate? i.e. the larger amount of data you write, the larger your stripe settings must be.

Those are the only 2 things I can think of without looking at your code.

Thanks,
Mohamad

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Matthieu Brucher
Sent: Tuesday, January 21, 2014 2:25 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] Performance hit when writing to big datasets

Hi,

I noticed today something strange. I've been using HDF5 with parallel processes, and when I raise the dataset size, the library seems to slow down.
Currently, I have a fixed number of processes writing data from my simulation. All these processes write 1MB concurrently to contiguous but exclusive locations in the 1D dataset. When having more than 1 billion elements (for instance), the library is getting really, really slow (more than 20 times slower than what a usual pwrite would take), where as nothing can be noticed when writing say 1 million cell.
Is there something fancy going on? I'm trying to access the simplest part of the library so that the H5write call can be more or less reduced to a pyrite call, but something is wrong. Should I set something to tell HF5 than it shouldn't do anything except than writing data from the calling process to the filesystem (Lustre in my case, MPI library configured with Lustre support).

Regards,

Matthieu Brucher
--
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

--
Information System Engineer, Ph.D.
Blog: http://matt.eifelle.com
LinkedIn: http://www.linkedin.com/in/matthieubrucher
Music band: http://liliejay.com/