Each file is a couple MB with ~1500 small datasets, of which only a small number will be read. The datasets are either 1- or 2-dimensional. Each dataset has at most a couple 1000 elements (mostly ints, some are floats or chars). I use the default VFD (sec2, right?). The datasets are contiguous. The specific results reported were on OS X, but I've seen a similar slowdown occur on Lustre as well (although not with this exact same revision of the code).
Here are my own "hdf5_hl" macros to read/write STL vector data:
#define HDF5CREATE_GROUP(loc_id, name) \
H5Gcreate(loc_id, name, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT)
#define HDF5OPEN_GROUP(loc_id, name) \
H5Gopen(loc_id, name, H5P_DEFAULT)
#define HDF5WRITE(group, ndims, dims, name, type, data) { \
hid_t dataspace = H5Screate_simple(ndims, dims, NULL), \
dataset = H5Dcreate(group, name, type, dataspace, \
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT); \
H5Dwrite(dataset, type, H5S_ALL, H5S_ALL, H5P_DEFAULT, &(data)[0]); \
H5Dclose(dataset); H5Sclose(dataspace); }
#define HDF5READ(group, ndims, name, type, data) { \
hid_t dataset = H5Dopen(group, name, H5P_DEFAULT), \
dataspace = H5Dget_space(dataset); \
hsize_t dims[2]; \
H5Sget_simple_extent_dims(dataspace,dims,NULL); \
data.resize(ndims==1 ? dims[0] : dims[0]*dims[1]); \
H5Dread(dataset, type, H5S_ALL, H5S_ALL, H5P_DEFAULT, &(data)[0]); \
H5Dclose(dataset); H5Sclose(dataspace); }
#define HDF5READ1(group, name, type, data) { \
hid_t dataset = H5Dopen(group, name, H5P_DEFAULT); \
H5Dread(dataset, type, H5S_ALL, H5S_ALL, H5P_DEFAULT, &(data)[0]); \
H5Dclose(dataset); }
···
On Jun 25, 2009, at 9:28 AM, Mark Howison wrote:
Hi Mark,
Can you give us some idea of the size of these files, the HDF5 VFD you
are using, the dimensionality of the datasets, and the storage method
(contig vs. chunk)? Also, I assume the slowdown you are reporting here
is not on a lustre, but on whatever filesystem you are using in OS X,
probably HFS+?
My guess is that mmap is performing large, contiguous IO operations,
while for some reason HDF5 isn't (possibly why you are hearing disk
thrashing).
Mark
On Thu, Jun 25, 2009 at 7:20 AM, Mark Moll<mmoll@cs.rice.edu> wrote:
[Just to summarize the discussion below: I have an MPI-based program that
needs parallel, asynchronous, read-only access to different data sets. No
two processes need access to the same data sets. In an old implementation
without HDF5 this is done by dividing the data sets into a number of files
that is a multiple of the number of processors and use mmap to access the
data in each file. This is rather inflexible, so I started to look at HDF5.
With HDF5 the program runs much slower.]
Initially, I thought the HDF5 version was much slower because all the data
sets were stored in one file, but now I see the same thing happening when I
put every data set into a separate file (# data sets >> # processors). I am
trying to figure out what's going on with Shark and ThreadViewer on OS X.
The profiles of the two versions of the program (with and without HDF5) look
similar, except for some system calls in the HDF5 version. I actually hear
the program hammering the disk in the HDF5 version, so I thought I'd use
ThreadViewer to see whether the program was blocked. The picture below shows
the difference between a thread in the HDF5 version (top) and the mmap-based
version (bottom). Light green means the thread is running, dark means
uninterruptible (usually system call), and yellow means recently running.
Clearly, there's a huge difference, but I don't know how to dig any deeper
to diagnose this issue.
I use OpenMPI 1.3.2, HDF5 1.8.3, and gcc 4.3.3 on OS X 10.5.7 (although I
have observed the same behavior on Ubuntu Linux and RHEL with Intel and PGI
compilers). I have also run the program through valgrind (on Linux and OS X)
to see if that would turn up any suspicious activity, but to no avail.
On Jun 23, 2009, at 9:24 AM, Quincey Koziol wrote:
Hi Mark,
On Jun 23, 2009, at 9:02 AM, Mark Moll wrote:
On Jun 23, 2009, at 8:53 AM, Quincey Koziol wrote:
On Jun 23, 2009, at 12:34 AM, Mark Howison wrote:
On Fri, Jun 19, 2009 at 9:47 AM, Mark Moll<mmoll@cs.rice.edu> >>>>>> wrote:
Just a few clarifications:
- The reading of data sets is mostly asynchronous; each node reach its
"own"
data sets.
Hmm, so each dataset belongs to only one node? It sounds like you
might not want to go the parallel HDF5 route. If the access is
asynchronous then you don't want to be using synchronized collective
calls.
Yes, this was my thought also. If the file is read-only, can
each process open it independently?
Right, this is what I did. However, multiple processes trying to read the
same file seems to introduce significantly more system overhead than
multiple processes reading different files.
Are you opening the file with MPI or with the default HDF5 file
driver?
It does sound more like a file system issue than something with
HDF5...
Quinceyfor
- File-per-node is indeed a problem with varying concurrency. That's
why I
was thinking of a data-set-per-file organization. Each file needs to
be
accessed by only one node. Each node would have to read many files.
The disadvantage to dataset-per-file is that it could lead to lots of
files. We have had problems in lustre with using basic filesystem
commands (ls, mv, cp, etc.) on large collections of small files (in a
recent example, 400K files totaling 230TB). Those commands will fail
with an error like "argument list too long..." Although, this may be a
limitation of those commands that isn't just specific to lustre.
I'm now experimenting with the data-set-per-file approach. I have an
index file that contains (among other things) the file location of each data
set. The data sets are stored in an automatically created directory
hierarchy, so that no directory will have too many files.
So the question is whether there is a performance difference between
parallel asynchronous reading of many files vs. one large file. I'm
guessing HDF5 does some preemptive fetching, which would work in favor
of
one large file.
I don't think that HDF5 will do any speculative read-aheads within a
file, if that is what you mean by preemptive fetching. It will only
read the regions you specify with a hyperslab selection.
Collective/parallel access only makes sense if you have many nodes
selecting different hyperslabs within the same dataset. If each node
is loading a different dataset, I think that collective access will
lead to unnecessary MPI communication, which will only become worse as
you scale up.
Thanks for clarifying. That was my impression, but I wanted to make sure
before giving up on that route.
--
Mark
--
Mark
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
--
Mark
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
