chunks of different sizes

Hdf-forum archives
1

Hi,

Is it possible to write chunks of different sizes? I mean, in a
parallel write, can chunks with different dimensions be written
collectively? For example, lets say that I have 4 processors and a
8-row by 6-column matrix. Mpi rank 0 writes from row 0 to 3 and column
0 to 3; Mpi rank 1 writes from 0 to 3 and 4 to 5; Mpi rank 2 writes
from 4 to 7 and 0 to 1; and finally Mpi rank 3 writes from 4 to 7 and
2 to 5. I am assuming C indexing. Notice that, although the processors
write chunks of different sizes, there is no overlap nor gaps. How can
I do that?

Thanks in advance,
Eduardo

2

Hi Eduardo,

···

On May 2, 2011, at 8:18 AM, Eduardo wrote:

Hi,

Is it possible to write chunks of different sizes? I mean, in a
parallel write, can chunks with different dimensions be written
collectively? For example, lets say that I have 4 processors and a
8-row by 6-column matrix. Mpi rank 0 writes from row 0 to 3 and column
0 to 3; Mpi rank 1 writes from 0 to 3 and 4 to 5; Mpi rank 2 writes
from 4 to 7 and 0 to 1; and finally Mpi rank 3 writes from 4 to 7 and
2 to 5. I am assuming C indexing. Notice that, although the processors
write chunks of different sizes, there is no overlap nor gaps. How can
I do that?

  I think you want to make different selections on different processes, not different chunk sizes. Try H5Sselect_hyperslab().

  Quincey

3

Is it possible to write chunks of different sizes? I mean, in a
parallel write, can chunks with different dimensions be written
collectively?

   I think you want to make different selections on different processes, not different chunk sizes\.  Try H5Sselect\_hyperslab\(\)\.

Different chunk sizes would be interesting to avoid IO contention when
the hyperslabs are not regular.

- Rhys

4

Different chunk sizes would be interesting to avoid IO contention when
the hyperslabs are not regular.

So, if my hyperslabs are not regular (similarly to my previous
example), does my application suffer IO contention? I mean, should I
avoid irregular hyperslabs, even if there is no overlap or gaps?

Best regards,
Eduardo

5

Hi Eduardo,

···

On May 3, 2011, at 12:36 PM, Eduardo wrote:

Different chunk sizes would be interesting to avoid IO contention when
the hyperslabs are not regular.

So, if my hyperslabs are not regular (similarly to my previous
example), does my application suffer IO contention? I mean, should I
avoid irregular hyperslabs, even if there is no overlap or gaps?

  Yes, independent access I/O performance will be best when there is a direct mapping of each process's selection to 1+ chunks and those chunks aren't accessed by other processes. Enabling collective I/O access may reduce or eliminate the penalty for accessing one chunk with multiple processes however.

  Quincey