Hi Ger,
A Friday 28 March 2008, Ger van Diepen escrigué:
Hi Francesc,
I realize we have very different needs and are in a way talking about
different things.
Indeed. There are many, many different scenarios when using a so
flexible library as HDF5.
You apply the chunk cache to pytables which is effectively a 1-dim
array of records. The size of each record can be different.
No, the case for PyTables is using the same record size for all the
entries in the table (this is in order to achieve maximum efficiency in
terms of data access and computations).
Usually access to it is linear or random.
That's correct.
You don't use the chunking mechanism of the datasets,
Not true. PyTables does use chunking extensively, most specially in
table datasets.
so you do not need to assemble the data
array from various chunks and can use a zero-copy strategy (provided
the endianness, etc. match).
Well, I realised that the example I gave yestarday was a bit simplistic,
and in fact I do need to assemble data in the pipeline data processing
(mainly for getting contiguous datasets and hence accelerating
computations). But inside PyTables itself there are situations where
this zero-copy strategy is actually used. My point is that a read-only
cache can accelerate things in *some* situations (but not always, of
course). And in these cases, this acceleration can be very beneficial
for an application.
The size of the chunk cache is hard to
predict for random access, so in a way any size may do and LRU is
usually the best mechanism.
Well, I don't grasp what you exactly mean here, but the main reason why
PyTables is using LRU caches internally is because it is normally used
interactively for data mining purposes and a frequent usage pattern is
to refine queries. For example, a typical sequence of queries in
PyTables could be:
1. [(row['lati'], row['long']) for row in table.where('pressure<200')]
2. [(row['lati'], row['long']) for row in table.where('(pressure>100) &
(pressure<200)']
3. [(row['lati'], row['long']) for row in table.where('((pressure<200) &
(pressure>100)) & ((temperature>10) & (temperature<15))']
So, it is out of doubt that a LRU cache for internal structures is our
best bet here.
On the other hand I want to store an N-dim array of fixed sized data
types (integer, real, complex) in a chunked way which requires that
when reading a data array it is assembled from the various chunks
(thus a lot of copying). Given an access pattern it is possible to
predict the optimal size of the cache. If the cache is too small, MRU
is the best way to deal with the cache because it will keep chunks in
the cache, while LRU will have removed the ones you are needing. E.g.
an array of 100x100 chunked in 10X10 chunks. You need 10 chunks when
accessing the array by vector (in x or y). If the cache can contain 9
chunks only, you can see that MRU can keep 8 or 9 chunks in the cache
while LRU will thrash.
I entirely agree that what is good for an application cannot be for
another, and you has given a good example where a LRU schema performs
really badly (but again, you always have the OS cache backing you up,
so the situation is not that bad ;-).
I think that the HDF5 chunking mechanism is primarily meant for the
latter case, but it seems that the chunk cache came in as some kind
of an afterthought (but I might be wrong). I think a single chunk
cache mechanism can serve both cases well, but might be somewhat
hard. Something to think about for Quincey.
I'm afraid it'll take quite a while before a more flexible caching
mechanism is added to HDF5. Until then HDF5 might be of limited use
to us.
Well, having a flexible caching in HDF5, supporting different methods
for different access patterns would be absolutely great. However, as
you already know, caching is a very tricky thing, and providing a
method suited for everyone's needs in a general library like HDF5
should represent a fair amount of work (this is more a matter of the
HDF5 crew, and the amount of resources they can afford to put at it,
but frankly, I'm not convinced that this would be a critical thing to
do, though).
IMHO, if a user has a desperate need for optimize some particular access
pattern that it is not efficiently implemented in HDF5, he should
carefully analyze it and come up with a sensible solution (be
implemented by himself or stolen from anywhere).
As an aside.
There is an image processing package called Karma (written by Richard
Gooch) using a different approach. It uses mmap to access a chunked
data array (so the OS takes care of caching). Getting array element
[k,j,i] is done as array[ xoffset[i] + yoffset[j] + zoffset[k] ]. For
any chunk size you can precalculate the 3 offset vectors. You can
even have chunks within chunks (outer chunks map to disk pages, inner
chunks to processor cache). This scheme is very useful when doing
array access in all dimensions around a pixel (e.g. to find the shape
of an object), but also works well for other access patterns. But
mmap has its limitations in file size, although that is not a problem
anymore on 64-bit systems.
I've also done some experiments with mmap, and frankly, I've found HDF5
to be far superior in terms of flexibility and performance (if
correctly used).
OTOH, letting the OS taking care of caching in an HDF5 app is as simple
as disabling internal chunk caching in HDF5. I don't know Karma, but
if you are using HDF5, you can still play with chunkshapes in order to
allow the OS filesystem cache to get the most from your access pattern.
However, as Quincey already said (and it is also my experience), you
should keep in mind that you can always expect more performance from a
application specific cache that from the OS cache. See, as for one,
the benchmaks that I send to this list yesterday, where PyTables Pro
internal cache can be up to more than 5x faster than the OS cache (15x
if using compression).
Finally, let me say that, in my experience, HDF5 is a *very* good piece
of software and, perhaps more importantly, very well maintained, which
is critical for data persistence apps. I've been very happy using it
for more than 5 years now and I've never found disappointed with it.
Of course, it is not perfect (nothing is), but do not underrate it
because, say, it doesn't have provision for your specific access
pattern. Complain, implement a new cache system, contribute it. But
be sure to ponder the complete bunch of features that HDF5 is offering
before deciding if it is for you or not.
Regards,
···
--
0,0< Francesc Altet http://www.carabos.com/
V V Cárabos Coop. V. Enjoy Data
"-"
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
I'll keep it in mind when I'm working on the chunk cache and see if I can find out what's going on.
It's just something to add to the HDF5 "wish list"...
What I was saying is that I'm considering