In this case i'm seeing a lot of 512 k writes, but I'd like to see
much bigger (at least 4 MB) writes. Do I want to adjust the
"vec_size" property, the "tconv_buf" property, or something else
entirely ?
You probably want to change the conversion buffer size with
H5Pset_buffer(). Unless you are generating very complex, disjoint
selections and then changes the vector size with
H5Pset_hyper_vector_size() might help. (Both of these are only
applicable to independent I/O, BTW)
Thanks for the pointers. I think this data type falls under "complex,
disjoint" when viewed on a per-process basis. The collective I/O
optimizations are so important that even though it's twice as much
data to write out doubles, the writes go much much faster. We'll see
what the tuning parameters can do.
Can I override HDF5's choice here and force it to take a collective
I/O path?
Unfortunately no, since there's a type conversion that needs to
occur. I suppose it's possible that each processes amount of
dataset is small enough that it could type convert into a buffer and
then perform a collective I/O operation, but that seems somewhat
unusual and we don't attempt to detect that currently.
We solve this in pnetcdf ... with lots of buffer copies :>
If the user-requested datatype is non-contiguous and we are doing a
write, we MPI_PACK() that type into a buffer, then MPI_UNPACK() that
buffer into a contiguous buffer. After that, we type-convert into a
third buffer and use that third buffer for the (collective) MPI write.
Reading is the reverse: collective read into a buffer, type convert,
MPI_Pack() into a buffer, MPI_Unpack() into the user's desired type.
Earlier this year we worked on something a little more sophisticated
and a bit more higher performing: Rob Ross extracted the MPI type
processing into a standalone library that things such as
parallel-netcdf, HDF5 and ROMIO can use to rip through MPI types more
efficiently and apply transformations on the data.
Now, I'm not suggesting we start re-working HDF5 type processing
tomorrow, but when the time comes, we've got options.
==rob
···
On Wed, Sep 09, 2009 at 04:27:41PM -0500, Quincey Koziol wrote:
On Sep 9, 2009, at 3:02 PM, Robert Latham wrote:
--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA