Quincey, Rob,
I was pondering an alternative (messy?) way of generating the multiple
group structure that we wanted and I thought I’d float this idea …
It goes like this.
Each process wants to create some arbitrary group/dataset, but in
general, each process is going to create a different group/dataset,
though it may be useful to have N processes in one group, or some other
combination (no need to be overly restrictive)
[nb. I’m approaching this from the vtkCompositeDataStructure - and in
particular vtkCompositeDataIterator - viewpoint if you’re familiar with
these structures]
Each process (collectively) calls HDF5CompositeCollectiveCreate(blah)
and passes in a tree object. The tree object contains nodes and leaves
which describe (in a format of my choosing at the moment) what groups,
subgroups datasets (dimensions etc) and everything else it wants to
create.
Because the call is collective, all processes participate, and
internally, the tree is populated by exchanging all-to-all information
about the nodes and leaves so that when complete, each process contains
the full tree, and not just the part it wanted to create.
The function returns an iterator object which each
process can now use to collectively
while not iterator->done
Create group/dataset etc
endwhile
Thus each process has knowledge about who is creating what, and all
create calls are done in the same order on each process, with all
required information so they return without blocking.
An iterator object (C++ only in my implementation I suppose) would
allow me to then do
iterator->local leaves only
loop
do stuff
endloop
when iterating over local datasets and not wanting to do stuff with
remote ones. (loop over all at start for create, loop over all at end
for close).
I am planning (probably) on implementing something along these lines as
a wrapper function on top of the core HDF5 as I believe it will
(elegantly?) solve my own problem. but I wonder if it is of interest
beyond my application - or if it has merits/flaws that provoke
discussion?
JB
···
-- John Biddiscombe, email:biddisco @ cscs.ch
CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07
Via Cantonale, 6928 Manno, Switzerland | Fax: +41 (91) 610.82.82
http://www.cscs.ch/
