I'm trying to maximize the way data is read from our filesystem, and
mainly this means reading from several readers chunks of 1MiB. My
problem is that sometime the data is not just one elementxnb cells,
but may be 2, 3, or 10 elements. This means that I may not be able to
properly read/write 1MiB if the data is stored as a (say) 2D array.
Is there a way to read from this array as if it were a 1D array?
Recomposing the data is not a problem for me as I has a mapping from
the data id to its location on each process.
If you turn on collective I/O it's more than likely the underlying
MPI-IO implementation will sort out the uneven distribution among
processes, coalesce small accesses into fewer bigger accesses, and
maybe even designate "aggregator" nodes to assist with scalability.
You'll get all this without needing to recompose the data.
==rob
···
On Fri, Oct 25, 2013 at 12:22:40PM +0200, Matthieu Brucher wrote:
Hi,
I'm trying to maximize the way data is read from our filesystem, and
mainly this means reading from several readers chunks of 1MiB. My
problem is that sometime the data is not just one elementxnb cells,
but may be 2, 3, or 10 elements. This means that I may not be able to
properly read/write 1MiB if the data is stored as a (say) 2D array.
Is there a way to read from this array as if it were a 1D array?
Recomposing the data is not a problem for me as I has a mapping from
the data id to its location on each process.
--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
All my tests with collective I/O indicate that I should not use it. As
a reference, my tests on collective I/O supposed that the data was
sorted properly before being written, and writing them from different
nodes or specific node (after local gathering) was 3 or 4 times slower
that writing raw files. With my new strategy, the data is sorted
properly (I don't even know how to tell HDF5 what pieces are where)
and written on the fly at almost maximum bandwidth.
Oh, in my test case, the data was almost properly distributed (1
million data per process)...
On Fri, Oct 25, 2013 at 12:22:40PM +0200, Matthieu Brucher wrote:
Hi,
I'm trying to maximize the way data is read from our filesystem, and
mainly this means reading from several readers chunks of 1MiB. My
problem is that sometime the data is not just one elementxnb cells,
but may be 2, 3, or 10 elements. This means that I may not be able to
properly read/write 1MiB if the data is stored as a (say) 2D array.
Is there a way to read from this array as if it were a 1D array?
Recomposing the data is not a problem for me as I has a mapping from
the data id to its location on each process.
If you turn on collective I/O it's more than likely the underlying
MPI-IO implementation will sort out the uneven distribution among
processes, coalesce small accesses into fewer bigger accesses, and
maybe even designate "aggregator" nodes to assist with scalability.
You'll get all this without needing to recompose the data.
==rob
--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
