Question about chunk cache

Hdf-forum archives
1

I've created a chunked dataset and created a chunk cache when writing to the
file. After closing the h5 file and reopening it. Do I have to recall
H5Pset_chunk_cache to set the appropriate chunk cache or when I call H5Dopen
the cache is automatically created for reading?

Best regards

···

--

Fernando Puech Helguero

---------------------------------------------------------------------
This message may contain confidential and/or privileged information.
If you are not the addressee or authorized to receive this for the
addressee, you must not use, copy, disclose or take any action based
on this message or any information herein. If you have received this
message by mistake, please advise the sender immediately by reply
e-mail and delete this message. Thank you for your cooperation.
Visit our web page: www.empre.es

Este mensaje puede contener informaci�n confidencial o privilegiada.
Si Vd. no es el destinatario ni est� autorizado por el mismo para
recibir este mensaje, Vd. no debe usar, copiar, revelar ni tomar
ninguna medida basada en este mensaje o en la informaci�n que
contiene. Si Vd. ha recibido este mensaje por error, notif�quelo de
forma inmediata al remitente por correo electr�nico y borre el
mensaje. Gracias por su cooperaci�n.
Visite nuestra p�gina web: www.empre.es
---------------------------------------------------------------------

Please, Do not print this message unless it is necessary.
Our environment is in our hands.
Antes de imprimir este mensaje, aseg�rese de que es necesario.
El medio ambiente est� en nuestra mano.

2
···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org > on behalf of Fernando Puech Helguero
fpe@empre.es

Sent: Friday, October 31, 2014 12:13 PM

To: hdf-forum@lists.hdfgroup.org

Subject: [Hdf-forum] Question about chunk cache

I’ve created a chunked dataset and created a chunk cache when writing to the file. After closing the h5 file and reopening it. Do I have to recall H5Pset_chunk_cache to set the appropriate
chunk cache or when I call H5Dopen the cache is automatically created for reading?

Best regards

–

Fernando Puech Helguero

Hi Fernando,

The chunk cache size is a property of accessing a dataset and is not stored with the dataset or the file. If you want to guarantee the same chunk cache settings each time
H5Dopen is called, you will need to call H5Dopen with a dataset access property list (H5Dopen(
hid_t loc_id, const char *name,
hid_t dapl_id )) where the chunk cache size is set by calling
H5Pset_chunk_cache for that property list. The property list can be used for multiple accesses in the same application.

For files where the same chunk cache size will be appropriate for all or most datasets, H5Pset_cache can be called with a file access property list to set the chunk cache
size for accessing all datasets in the file.

Both methods can be used in combination, in which case the chunk cache size set by H5Pset_cache will apply except for specific datasets where H5Dopen is called with dataset
property list with the chunk cache size set by H5Pset_chunk_cache.

In the absence of any cache settings, H5Dopen will by default create a 1 MB chunk cache for the opened dataset. If this size happens to be appropriate you don’t have to
call either function to set the chunk cache size.

It is also possible that a change in access pattern for later access to a dataset will change the appropriate chunk cache size. For more detailed information on chunks,
chunk caching, and performance pitfalls please see “Chunking in HDF5” at htttp://www.hdfgroup.org/HDF5/doc/Advanced/Chunking/index.html.

Larry Knox

  • This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for
    the addressee, you must not use, copy, disclose or take any action based on this message or any information herein. If you have received this message by mistake, please advise the sender immediately by reply e-mail and delete this message. Thank you for your
    cooperation. Visit our Web page: www.empre.es.*

  • Este mensaje puede contener información confidencial o privilegiada. Si Vd. no es el destinatario ni está autorizado por el mismo para recibir este
    mensaje, Vd. no debe usar, copiar, revelar ni tomar ninguna medida basada en este mensaje o en la información que contiene. Si Vd. ha recibido este mensaje por error, notifíquelo de forma inmediata al remitente por correo electrónico y borre el mensaje. Gracias
    por su cooperación.Visite nuestra página web: www.empre.es.*

Please, Do not print this message unless it is necessary.Our environment is in our hands.

Antes de imprimir este mensaje, asegúrese de que es necesario. El medio ambiente está en nuestra mano.

3

Hi Larry,

Thank you very much for the explanation, because it wasn’t clear in the
documentation.

···

--

Fernando Puech Helguero

De: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] En nombre de
Larry Knox
Enviado el: martes, 04 de noviembre de 2014 16:33
Para: hdf-forum@lists.hdfgroup.org
Asunto: Re: [Hdf-forum] Question about chunk cache

  _____

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Fernando
Puech Helguero <fpe@empre.es>
Sent: Friday, October 31, 2014 12:13 PM
To: hdf-forum@lists.hdfgroup.org
Subject: [Hdf-forum] Question about chunk cache

I’ve created a chunked dataset and created a chunk cache when writing to the
file. After closing the h5 file and reopening it. Do I have to recall
H5Pset_chunk_cache to set the appropriate chunk cache or when I call H5Dopen
the cache is automatically created for reading?

Best regards

--

Fernando Puech Helguero

Hi Fernando,

The chunk cache size is a property of accessing a dataset and is not stored
with the dataset or the file. If you want to guarantee the same chunk cache
settings each time H5Dopen is called, you will need to call H5Dopen with a
dataset access property list (H5Dopen( hid_t loc_id, const char *name, hid_t
dapl_id )) where the chunk cache size is set by calling H5Pset_chunk_cache
for that property list. The property list can be used for multiple accesses
in the same application.

For files where the same chunk cache size will be appropriate for all or
most datasets, H5Pset_cache can be called with a file access property list
to set the chunk cache size for accessing all datasets in the file.

Both methods can be used in combination, in which case the chunk cache size
set by H5Pset_cache will apply except for specific datasets where H5Dopen is
called with dataset property list with the chunk cache size set by
H5Pset_chunk_cache.

In the absence of any cache settings, H5Dopen will by default create a 1 MB
chunk cache for the opened dataset. If this size happens to be appropriate
you don’t have to call either function to set the chunk cache size.

It is also possible that a change in access pattern for later access to a
dataset will change the appropriate chunk cache size. For more detailed
information on chunks, chunk caching, and performance pitfalls please see
“Chunking in HDF5” at
htttp://www.hdfgroup.org/HDF5/doc/Advanced/Chunking/index.html.

Larry Knox

  _____

This message may contain confidential and/or privileged information. If you
are not the addressee or authorized to receive this for the addressee, you
must not use, copy, disclose or take any action based on this message or any
information herein. If you have received this message by mistake, please
advise the sender immediately by reply e-mail and delete this message. Thank
you for your cooperation. Visit our Web page: www.empre.es.
<http://www.empre.es>

Este mensaje puede contener información confidencial o privilegiada. Si Vd.
no es el destinatario ni está autorizado por el mismo para recibir este
mensaje, Vd. no debe usar, copiar, revelar ni tomar ninguna medida basada en
este mensaje o en la información que contiene. Si Vd. ha recibido este
mensaje por error, notifíquelo de forma inmediata al remitente por correo
electrónico y borre el mensaje. Gracias por su cooperación.Visite nuestra
página web: www.empre.es. <http://www.empre.es>

  _____

Please, Do not print this message unless it is necessary.Our environment is
in our hands.

Antes de imprimir este mensaje, asegúrese de que es necesario. El medio
ambiente está en nuestra mano.

---------------------------------------------------------------------
This message may contain confidential and/or privileged information.
If you are not the addressee or authorized to receive this for the
addressee, you must not use, copy, disclose or take any action based
on this message or any information herein. If you have received this
message by mistake, please advise the sender immediately by reply
e-mail and delete this message. Thank you for your cooperation.
Visit our web page: www.empre.es

Este mensaje puede contener información confidencial o privilegiada.
Si Vd. no es el destinatario ni está autorizado por el mismo para
recibir este mensaje, Vd. no debe usar, copiar, revelar ni tomar
ninguna medida basada en este mensaje o en la información que
contiene. Si Vd. ha recibido este mensaje por error, notifíquelo de
forma inmediata al remitente por correo electrónico y borre el
mensaje. Gracias por su cooperación.
Visite nuestra página web: www.empre.es
---------------------------------------------------------------------

Please, Do not print this message unless it is necessary.
Our environment is in our hands.
Antes de imprimir este mensaje, asegúrese de que es necesario.
El medio ambiente está en nuestra mano.