Hi,
I'm interested in being able to change the cache settings for chunks in HDF5.
For this, I plan to call:
{{{
herr_t H5Pset_cache(hid_t plist_id, int mdc_nelmts, size_t rdcc_nelmts,
size_t rdcc_nbytes, double rdcc_w0)
}}}
before opening an HDF5 file.
Most of the parameters of the H5Pset_cache call are clear, but I cannot
determine a good default for rdcc_nelemts. From the manual:
{{{
The number of elements (objects) in the raw data chunk cache is rdcc_nelmts.
The total size of the raw data chunk cache and the preemption policy are
rdcc_nbytes and rdcc_w0, respectively.
}}}
I'm choosing 10 by default at this time, but I'd be grateful if anybody can
tell me whether this is a decent value or not (and the rational behind this).
Cheers,
···
--
Francesc Alted
A Monday 21 December 2009 13:27:36 Francesc Alted escrigué:
Hi,
I'm interested in being able to change the cache settings for chunks in
HDF5. For this, I plan to call:
{{{
herr_t H5Pset_cache(hid_t plist_id, int mdc_nelmts, size_t rdcc_nelmts,
size_t rdcc_nbytes, double rdcc_w0)
}}}
before opening an HDF5 file.
Most of the parameters of the H5Pset_cache call are clear, but I cannot
determine a good default for rdcc_nelemts. From the manual:
{{{
The number of elements (objects) in the raw data chunk cache is
rdcc_nelmts. The total size of the raw data chunk cache and the preemption
policy are rdcc_nbytes and rdcc_w0, respectively.
}}}
I'm choosing 10 by default at this time, but I'd be grateful if anybody can
tell me whether this is a decent value or not (and the rational behind
this).
Ok. After considering this for a while, I'll stick with a value of 256 as the
default for rdcc_nelmts. I hope it would not be too much --although, frankly,
the rdcc_nbytes will force a more lower number generally speaking.
Cheers,
···
--
Francesc Alted
Francesc,
This may be a bit late, but have you looked at the documentation for H5Pset_chunk_cache? The parameters there are the same, but the descriptions are more thorough. Due to the (simplistic) hashing algorithm, rdcc_nelmts should be a prime number and should probably be larger than 256 - the default in HDF5 is 521 (not 512) and I believe that is too small in many cases. There is very little downside the setting a larger hash table.
Thanks,
-Neil
···
On 12/22/2009 02:14 AM, Francesc Alted wrote:
A Monday 21 December 2009 13:27:36 Francesc Alted escrigu�:
Hi,
I'm interested in being able to change the cache settings for chunks in
HDF5. For this, I plan to call:
{{{
herr_t H5Pset_cache(hid_t plist_id, int mdc_nelmts, size_t rdcc_nelmts,
size_t rdcc_nbytes, double rdcc_w0)
}}}
before opening an HDF5 file.
Most of the parameters of the H5Pset_cache call are clear, but I cannot
determine a good default for rdcc_nelemts. From the manual:
{{{
The number of elements (objects) in the raw data chunk cache is
rdcc_nelmts. The total size of the raw data chunk cache and the preemption
policy are rdcc_nbytes and rdcc_w0, respectively.
}}}
I'm choosing 10 by default at this time, but I'd be grateful if anybody can
tell me whether this is a decent value or not (and the rational behind
this).
Ok. After considering this for a while, I'll stick with a value of 256 as the
default for rdcc_nelmts. I hope it would not be too much --although, frankly,
the rdcc_nbytes will force a more lower number generally speaking.
Cheers,
A Tuesday 22 December 2009 16:31:50 Neil Fortner escrigué:
Francesc,
This may be a bit late, but have you looked at the documentation for
H5Pset_chunk_cache? The parameters there are the same, but the
descriptions are more thorough. Due to the (simplistic) hashing
algorithm, rdcc_nelmts should be a prime number and should probably be
larger than 256 - the default in HDF5 is 521 (not 512) and I believe
that is too small in many cases. There is very little downside the
setting a larger hash table.
Thanks Neil. Yes, H5Pset_chunk_cache() doc is much better (frankly, I did not
know it). Next time will tune this number further. I plan to do some
benchmarks for some common use cases and set this number accordingly.
···
--
Francesc Alted
