Hi Quincey,
Attached you find a program that can be used to write chunked data and read it back in various ways.
For example:
tHDF5 1024 1024 10 32 32 2
creates the file and dataset and fills it chunk by chunk. The first three arguments give the cube shape, the latter the chunk shape. Note that the chunk shape must fit integrally in the cube shape (makes the test program easier).
The 7th argument can be used to specify how to read it back.
tHDF5 1024 1024 10 32 32 2 t
reads the dataset chunk by chunk.
tHDF5 1024 1024 10 32 32 2 x
reads the dataset by x-vectors. Similarly use y and z to read by y- or z-vector.
In this example reading tile by tile is fast, by x or y is slower, while by z is very slow. The smaller the hyperslab, the slower HDF5 gets.
The real code gets the cube and chunk shape from the dataset, but in this test I was lazy and it must always be specified on the command line (and match the shapes used when creating the dataset).
It might be that something stupid is done in this code. If so, please tell me.
I was wondering if testing if a chunk is in the cache might be the culprit. For each hyperslab access HDF5 needs to test if the required chunks are in the cache. It means hashing the chunk number and looking it up in the hash table. I can imagine that can take a long time if it has to be done millions of time, but doubt it. 100 seconds for such operations seems too much.
Cheers,
Ger van Diepen
tHDF5.cc (13.2 KB)
Hi Ger,
Hi Quincey,
Attached you find a program that can be used to write chunked data and read it back in various ways.
For example:
tHDF5 1024 1024 10 32 32 2
creates the file and dataset and fills it chunk by chunk. The first three arguments give the cube shape, the latter the chunk shape. Note that the chunk shape must fit integrally in the cube shape (makes the test program easier).
The 7th argument can be used to specify how to read it back.
tHDF5 1024 1024 10 32 32 2 t
reads the dataset chunk by chunk.
tHDF5 1024 1024 10 32 32 2 x
reads the dataset by x-vectors. Similarly use y and z to read by y- or z-vector.
In this example reading tile by tile is fast, by x or y is slower, while by z is very slow. The smaller the hyperslab, the slower HDF5 gets.
The real code gets the cube and chunk shape from the dataset, but in this test I was lazy and it must always be specified on the command line (and match the shapes used when creating the dataset).
It might be that something stupid is done in this code. If so, please tell me.
I've looked through your code and I think it's sensible. I'll file a bug report and we'll try to get some time to work on it.
Thanks,
Quincey
···
On Jun 25, 2009, at 8:20 AM, Ger van Diepen wrote:
I was wondering if testing if a chunk is in the cache might be the culprit. For each hyperslab access HDF5 needs to test if the required chunks are in the cache. It means hashing the chunk number and looking it up in the hash table. I can imagine that can take a long time if it has to be done millions of time, but doubt it. 100 seconds for such operations seems too much.
Cheers,
Ger van Diepen
<tHDF5.cc>----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
