Hi Quincey,
A Thursday 27 September 2007, Quincey Koziol escrigué:
> As you can see, most of the times for the libraries are more or
> less the
> same (remember, counts are not exact), except for HDF5 itself which
> takes around 5x more that the original version. Also noticeable is
> that the kernel spends here a 7% more as well (I don't know if this
> is significant, probably not).
Hmm, yes, that's a bit more than I'd expect, although if you are
doing complicated selections, having the extra time in the HDF5
routines might be reasonable.
Yeah, for the general purposes I guess so. It's just that I wanted to
give logical combinations of chunks a chance for speeding-up my own
stuff 
> If we look into the details of the routines consuming more cpu
> inside HDF5, this is what we get for the original version:
>
> samples % symbol name
> 417 68.3607 H5Z_filter_shuffle
> 10 1.6393 .plt
> 8 1.3115 H5D_create_chunk_map
> 8 1.3115 H5S_select_hyperslab
> 6 0.9836 H5D_istore_cmp3
> 6 0.9836 H5FD_sec2_read
> 6 0.9836 H5T_cmp
> 5 0.8197 H5B_find
> 5 0.8197 H5C_protect
> 5 0.8197 H5C_unprotect
> 5 0.8197 H5D_istore_lock
> 5 0.8197 H5FL_reg_malloc
Hmm, I wonder why the shuffle filter is taking so much CPU time...
:-?
I'll tell you why: I work with pretty large chunksizes (64 KB in this
example), with compound elements (24 bytes each). I've noticed that
shuffle times increases quite a lot with the chunksize (which is
expected anyway).
The __udivdi3 and __umoddi3 routines are for 64-bit integer division
and 'mod' operations. Computing the offsets for "complicated"
hyperslab selections involves some division & mod operations, so it's
reasonable to see them in the profile, although I do think it's
possible there's a problem with the algorithm that they are consuming
so much CPU time.
Can you describe the hyperslab operations you are performing in
order to generate the profile above? Also, if you could translate a
sample program showing the problem from Python into C and send it to
me, that would make it easier for me to work on speedups/fixes at
this end.
My setup is pretty simple: it reads 5000 chunks (64 KB in size as said
before) randomly chosen from a compound (24 bytes/element) dataset (100
Melements, for a total of 2.4 GB) using compression (225% of
efficiency, for a grand total of 1.1 GB). Also, the access order is
sorted so that the disk heads doesn't have to travel too much going
from one cylinder to the other. Another optimization was that, for
reading puposes, the chunks are grouped in buffers of 8 chunks, and if
two or more chunks were continguous, these are joined into a bigger
hyperslice for issuing a single I/O request (but this kind of joins
only happens in about the 10% of the chunks, so you can disregard this
optimization for your benchmarking purposes).
For your reference, here you have the basic loop routine that builds the
dataspace with the 8 OR'ed chunks to be read:
/* ****************************** */
/* Fill the dataset with the hyperslices to be read */
nrecout = 0;
for (n=0; n < nhyperslices; n++) {
/* Define a hyperslab in the dataset of the size of the records */
offset[0] = starts[n];
count[0] = nrecords[n];
H5Sselect_hyperslab(space_id, (n==0 ? H5S_SELECT_SET :
H5S_SELECT_OR), offset, NULL, count, NULL);
nrecout += nrecords[n];
}
/* ****************************** */
I'm a bit swamped right now, so I won't be able to provide a sample
program in C anytime soon. I'll try to do this after I finished my
current duties if you still need it.
At any rate, I'd be more interested (and I guess you will as well) in
knowing if it is possible to pack an N-chunk sparse read in a single
shot request to the operating system, than optimizing the CPU times.
As you can guess, the bottleneck with this kind of reads is more the
seek times than the CPU overhead, and this will become more and more
important as the time goes (CPU's are more and powerful each year, but
disk seek times are still barely the same now than 10 years ago).
Unfortunately, it is bad news that Posix doesn't seem to provide an
easy solution for this.
> [1] http://oprofile.sourceforge.net
Nice tool for profiling - thanks for the pointer. 
It works pretty well, and use it quite a lot. Besides, it's free
···
--
0,0< Francesc Altet http://www.carabos.com/
V V Cárabos Coop. V. Enjoy Data
"-"
