I would suggest trying the enhancements that come with using the
latest version of the file format, which can be enabled by calling
H5Pset_libver_bounds() with both bounds settings to
H5F_LIBVER_LATEST.
Thanks. I tried this, but the difference is minimal, for my test
program.
You can nest HDF5's variable-length datatypes arbitrarily deep - does
that give you what you are looking for?
One of the issues is that the data won't fit into memory. Worst
case, the entire pile of data is 10-100+ terabytes in size.
I'm now trying plan B: concatenating the original datasets,
end-to-end, in one big dataset. I have another dataset that keeps track
of the locations and sizes of the original datasets, in the big dataset.
So far, this seems to be somewhat fast and scalable (the write times
seem to scale roughly linearly). I need to do read timings, though.
On a different note: does h5dump support split files? I can't seem
to get h5dump to recognize them. I wrote some test code that uses the
split driver to write two files:
ext.h5.meta
ext.h5.raw
$ h5dump ext.h5.meta
h5dump error: unable to open file "ext.h5.meta"
$ h5dump ext.h5.raw
h5dump error: unable to open file "ext.h5.raw"
$ h5dump --filedriver split ext.h5.meta
h5dump error: unable to open file "ext.h5.meta"
$ h5dump --filedriver split ext.h5.raw
h5dump error: unable to open file "ext.h5.raw"
I don't think it's a problem with the test code, as it produces a
dumpable "ext.h5" file if I comment out the call to H5Pset_fapl_split()
(the test code originated from the h5_extend.c example).
···
--
Darryl Okahata
darrylo@soco.agilent.com
DISCLAIMER: this message is the author's personal opinion and does not
constitute the support, opinion, or policy of Agilent Technologies, or
of the little green men that have been following him all day.