I am about to implement a parallel writer using the HDF5 C-API.
The data I need to write is distributed over the different partitions in contiguous memory (c arrays). To keep things simple let the rank be 1 (1D/vector). Each process’ array has a different size (small deviation, less 10%).
So the data is irregular in just one dimension.
The data can be sparse or dense, for the sparse case I wish to use compression (chunking implied).
From this example I started experimenting and managed to have compression/chunking with same sized arrays. Without chunking I even got different sized arrays running.
Because I am using collective write I assume that the chunk size must be equal to all processes. The whole data may fit into memory, so that the chunk size could cover the whole data set.
Could you please provide some links to examples with chunking and collective writes and give hints how that could be adapted to different sized arrays?