[Pytables-users] Writing to a dataset with 'wrong' chunksize

Hi Francesc,

A Monday 03 December 2007, Francesc Altet escrigué:

Ups, I've ended with a similar program and send it to the
hdf-forum@hdfgroup.org list past Saturday. I'm attaching my own
version (which is pretty similar to yours). Sorry for not sending
you a copy of my previous message, because it could saved you some
work :-/

Well, as Ivan pointed out, a couple of glitches slipped in my program.
I'm attaching the correct version, but the result is the same, i.e.
when N=600. I'm getting a segfault both under HDF5 1.6.5 and 1.8.0
beta5.

  I was able to duplicate the segfault with your program, but it was a stack overflow and if you move the "data" array out of main() and make it a global variable, things run to completion without error. It's _really_ slow and chews _lots_ of memory still (because you are slicing the dataset the "wrong" way), but everything seems to be working correctly.

  It's somewhat hard to fix the "slicing the wrong way" problem, because the library is building a list of all the chunks that will be affected by each I/O operation (so that we can do all the I/O on each chunk at once) and that has some memory issues when dealing with I/O operations that affect so many chunks at once right now. Building a list of all the affected chunks is good for the parallel I/O case, but could be avoided in the serial I/O case, I think. However, that would probably make the code difficult to maintain... :-/

  You could try adjusting the chunk cache size larger, which would probably help, if you make it large enough to hold all the chunks for the dataset.

  Quincey

···

On Dec 3, 2007, at 11:21 AM, Francesc Altet wrote:

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Fransesc et al:
Just to elaborate a little bit on Quincey's "slicing the wrong way" explanation. (I hope I'm not just confusing matters.)

If possible you want to design the shape of the chunk so that you get the most useful data with the fewest number of accesses. If accesses are mostly contiguous elements along a certain dimension, you shape the chunk to contain the most elements along that dimension. If accesses are random shapes and sizes, then it gets a little tricky -- we just generally recommend a square (cube, etc.), but that may not be as good as, say, a shape that has the same proportions as your dataset.

So, for instance if your dataset is 3,000x6,000 (3,000 rows, 6,000 columns) and you always access a single column, then each chunk should contain as much of a column as possible, given your best chunk size. If we assume a good chunk size is 600 elements, then your chunks would all be 600x1, and accessing any column in its entirety would take 10 accesses. Having each chunk be a part of a row (1x600) would give you the worst performance in this case, since you'd need to access 6,000 chunks to access a column.

If accesses are unpredictable, perhaps a chunk size of 30x60 would be best, as your worst case performance (for reading a single column or row) would take 100 accesses. (By worst case, I'm thinking of the case where you have to do the most accesses per useful data element.)

In other cases, such as when you slice it one way you don't care about performance, but when you slice it another way you really do, would call for a chunk to be shaped accordingly.

Mike

···

At 11:01 AM 12/4/2007, Quincey Koziol wrote:

Hi Francesc,

On Dec 3, 2007, at 11:21 AM, Francesc Altet wrote:

A Monday 03 December 2007, Francesc Altet escrigué:

Ups, I've ended with a similar program and send it to the
hdf-forum@hdfgroup.org list past Saturday. I'm attaching my own
version (which is pretty similar to yours). Sorry for not sending
you a copy of my previous message, because it could saved you some
work :-/

Well, as Ivan pointed out, a couple of glitches slipped in my program.
I'm attaching the correct version, but the result is the same, i.e.
when N=600. I'm getting a segfault both under HDF5 1.6.5 and 1.8.0
beta5.

        I was able to duplicate the segfault with your program, but it was a
stack overflow and if you move the "data" array out of main() and
make it a global variable, things run to completion without error.
It's _really_ slow and chews _lots_ of memory still (because you are
slicing the dataset the "wrong" way), but everything seems to be
working correctly.

        It's somewhat hard to fix the "slicing the wrong way" problem,
because the library is building a list of all the chunks that will be
affected by each I/O operation (so that we can do all the I/O on each
chunk at once) and that has some memory issues when dealing with I/O
operations that affect so many chunks at once right now. Building a
list of all the affected chunks is good for the parallel I/O case,
but could be avoided in the serial I/O case, I think. However, that
would probably make the code difficult to maintain... :-/

        You could try adjusting the chunk cache size larger, which would
probably help, if you make it large enough to hold all the chunks for
the dataset.

        Quincey

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

--
Mike Folk The HDF Group http://hdfgroup.org 217.244.0647
1901 So. First St., Suite C-2, Champaign IL 61820

Hi Mike and others,

Sorry for the delay answering, but I was traveling past week.

Thanks for your explanation. I understand what you both are saying, and
this reveals how important is choosing a correct chunkshape when you
want to get decent performance in HDF5 I/O.

PyTables initially tried to hide such 'low level' details to the user,
but after realising how important is this, we introduced
the 'chunkshape' parameter in dataset constructors in the 2.0 series.
While PyTables still tries hard to avoid users to think about
chunkshape issues and automatically compute 'optimal' chunksizes, the
fact is that this only works well when the user wants to access their
data in the so-called 'C-order' (i.e. data is arranged in rows, not
columns).

However, users may have many valid reasons to choose another
arrangements than the C-order one. So, in order to cope with this, I'm
afraid that the only solution will be to add a specific section in the
PyTables User's Guide in order to carefully explain this. Your
explanations will definitely help to build a better guide on how to
choose the chunkshape that best fits the needs of the users.

Thanks!

A Thursday 06 December 2007, Mike Folk escrigué:

···

Fransesc et al:
Just to elaborate a little bit on Quincey's
"slicing the wrong way" explanation. (I hope I'm not just confusing
matters.)

If possible you want to design the shape of the
chunk so that you get the most useful data with
the fewest number of accesses. If accesses are
mostly contiguous elements along a certain
dimension, you shape the chunk to contain the
most elements along that dimension. If accesses
are random shapes and sizes, then it gets a
little tricky -- we just generally recommend a
square (cube, etc.), but that may not be as good
as, say, a shape that has the same proportions as your dataset.

So, for instance if your dataset is 3,000x6,000
(3,000 rows, 6,000 columns) and you always access
a single column, then each chunk should contain
as much of a column as possible, given your best
chunk size. If we assume a good chunk size is
600 elements, then your chunks would all be
600x1, and accessing any column in its entirety
would take 10 accesses. Having each chunk be a
part of a row (1x600) would give you the worst
performance in this case, since you'd need to
access 6,000 chunks to access a column.

If accesses are unpredictable, perhaps a chunk
size of 30x60 would be best, as your worst case
performance (for reading a single column or row)
would take 100 accesses. (By worst case, I'm
thinking of the case where you have to do the
most accesses per useful data element.)

In other cases, such as when you slice it one way
you don't care about performance, but when you
slice it another way you really do, would call
for a chunk to be shaped accordingly.

Mike

At 11:01 AM 12/4/2007, Quincey Koziol wrote:
>Hi Francesc,
>
>On Dec 3, 2007, at 11:21 AM, Francesc Altet wrote:
>>A Monday 03 December 2007, Francesc Altet escrigué:
>>>Ups, I've ended with a similar program and send it to the
>>>hdf-forum@hdfgroup.org list past Saturday. I'm attaching my own
>>>version (which is pretty similar to yours). Sorry for not sending
>>>you a copy of my previous message, because it could saved you some
>>>work :-/
>>
>>Well, as Ivan pointed out, a couple of glitches slipped in my
>> program. I'm attaching the correct version, but the result is the
>> same, i.e. when N=600. I'm getting a segfault both under HDF5
>> 1.6.5 and 1.8.0 beta5.
>
> I was able to duplicate the segfault
> with your program, but it was a
>stack overflow and if you move the "data" array out of main() and
>make it a global variable, things run to completion without error.
>It's _really_ slow and chews _lots_ of memory still (because you are
>slicing the dataset the "wrong" way), but everything seems to be
>working correctly.
>
> It's somewhat hard to fix the "slicing the wrong way"
> problem, because the library is building a list of all the chunks
> that will be affected by each I/O operation (so that we can do all
> the I/O on each chunk at once) and that has some memory issues when
> dealing with I/O operations that affect so many chunks at once
> right now. Building a list of all the affected chunks is good for
> the parallel I/O case, but could be avoided in the serial I/O case,
> I think. However, that would probably make the code difficult to
> maintain... :-/
>
> You could try adjusting the chunk cache size larger, which
> would probably help, if you make it large enough to hold all the
> chunks for the dataset.
>
> Quincey
>
>
>--------------------------------------------------------------------
>-- This mailing list is for HDF software users discussion.
>To subscribe to this list, send a message to
> hdf-forum-subscribe@hdfgroup.org. To unsubscribe, send a message to
> hdf-forum-unsubscribe@hdfgroup.org.

--
Mike Folk The HDF Group http://hdfgroup.org 217.244.0647
1901 So. First St., Suite C-2, Champaign IL 61820

---------------------------------------------------------------------
- This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
hdf-forum-subscribe@hdfgroup.org. To unsubscribe, send a message to
hdf-forum-unsubscribe@hdfgroup.org.

--

0,0< Francesc Altet http://www.carabos.com/

V V Cárabos Coop. V. Enjoy Data
"-"

This is particularly true in high resolution electron microscopy.
As the resolution improves sometimes it becomes necessary to change the order of axes to match xray datasets.
It would be preferable not to rewrite a very large dataset because you realize it was generated as a left handed coordinate system instead of right handed.
But the operative issue is the performance hit.

···

-----Original Message-----
From: Francesc Altet [mailto:faltet@carabos.com]
Sent: Wed 12/12/2007 4:13 AM
To: Mike Folk
Cc: hdf-forum@hdfgroup.org; Quincey Koziol; pytables-users@lists.sourceforge.net; Pauli Virtanen
Subject: Re: [Pytables-users] Writing to a dataset with 'wrong' chunksize

However, users may have many valid reasons to choose another
arrangements than the C-order one.

If I had a situation where I create a very large chunked dataset that had xyz ordering, and later find out that the preferred readout should be yzx;
then what would the best performance solution be:

1) recreate a new dataset with the yzx ordering
2) keep the old dataset and write software to reorganize the data
3) not worry about it because the performance hit is small because the data can be quickly reorganized as it is coming off storage into ram.
4) keep the old dataset, but coding changes deep in HDF will be needed to achieve good performance.

Matthew Dougherty
713-433-3849
National Center for Macromolecular Imaging
Baylor College of Medicine/Houston Texas USA

···

=========================================================================

-----Original Message-----
From: Dougherty, Matthew T. [mailto:matthewd@bcm.edu]
Sent: Wed 12/12/2007 12:48 PM
To: Francesc Altet; Mike Folk
Cc: hdf-forum@hdfgroup.org; Quincey Koziol; pytables-users@lists.sourceforge.net; Pauli Virtanen
Subject: RE: [Pytables-users] Writing to a dataset with 'wrong' chunksize

This is particularly true in high resolution electron microscopy.
As the resolution improves sometimes it becomes necessary to change the order of axes to match xray datasets.
It would be preferable not to rewrite a very large dataset because you realize it was generated as a left handed coordinate system instead of right handed.
But the operative issue is the performance hit.

-----Original Message-----
From: Francesc Altet [mailto:faltet@carabos.com]
Sent: Wed 12/12/2007 4:13 AM
To: Mike Folk
Cc: hdf-forum@hdfgroup.org; Quincey Koziol; pytables-users@lists.sourceforge.net; Pauli Virtanen
Subject: Re: [Pytables-users] Writing to a dataset with 'wrong' chunksize

However, users may have many valid reasons to choose another
arrangements than the C-order one.

A Thursday 20 December 2007, Dougherty, Matthew T. escrigué:

If I had a situation where I create a very large chunked dataset that
had xyz ordering, and later find out that the preferred readout
should be yzx; then what would the best performance solution be:

1) recreate a new dataset with the yzx ordering
2) keep the old dataset and write software to reorganize the data
3) not worry about it because the performance hit is small because
the data can be quickly reorganized as it is coming off storage into
ram. 4) keep the old dataset, but coding changes deep in HDF will be
needed to achieve good performance.

Well, I'd say that this largely depends on what kind of read performance
would you expect to obtain. If, as you said, your dataset is very
large, then the chunks should be large as well, and reading a whole
chunk could be very costly in terms of disk I/O effort but also in
terms of CPU if your dataset is compressed. So, if I were using
compression and wanted to obtain maximum performance, I'd choose option
1) with no discussion, as there is very little that the software can do
to avoid the necessary disk/CPU resources consumption that implies
an 'out-of-order' read.

This is in the case that you are using compression. However,
experiments that I've carried some time ago tend to show that a chunk
has not necessarily to be read completely as long it is not compressed,
i.e. partial chunk reads are possible (Quincey, correct me if I'm
wrong). In that case, perhaps options 3) and 4) could be feasible too.

Finally, I don't understand well the difference between 1) and 2), but
if the only difference is that 2) implies having two different datasets
on disk simultaneously, then 1) is clearly better, IMO.

My 2 cents,

···

--

0,0< Francesc Altet http://www.carabos.com/

V V Cárabos Coop. V. Enjoy Data
"-"

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

A Thursday 20 December 2007, Dougherty, Matthew T. escrigué:

If I had a situation where I create a very large chunked dataset that
had xyz ordering, and later find out that the preferred readout
should be yzx; then what would the best performance solution be:

1) recreate a new dataset with the yzx ordering
2) keep the old dataset and write software to reorganize the data
3) not worry about it because the performance hit is small because
the data can be quickly reorganized as it is coming off storage into
ram. 4) keep the old dataset, but coding changes deep in HDF will be
needed to achieve good performance.

Well, I'd say that this largely depends on what kind of read performance
would you expect to obtain. If, as you said, your dataset is very
large, then the chunks should be large as well, and reading a whole
chunk could be very costly in terms of disk I/O effort but also in
terms of CPU if your dataset is compressed. So, if I were using
compression and wanted to obtain maximum performance, I'd choose option
1) with no discussion, as there is very little that the software can do
to avoid the necessary disk/CPU resources consumption that implies
an 'out-of-order' read.

This is in the case that you are using compression. However,
experiments that I've carried some time ago tend to show that a chunk
has not necessarily to be read completely as long it is not compressed,
i.e. partial chunk reads are possible (Quincey, correct me if I'm
wrong). In that case, perhaps options 3) and 4) could be feasible too.

Finally, I don't understand well the difference between 1) and 2), but
if the only difference is that 2) implies having two different datasets
on disk simultaneously, then 1) is clearly better, IMO.

  Frankly, it could be any of the options above, depending on your performance requirements, etc. HDF5 currently (in the 1.6.x & 1.8.x branches) examines the file and memory dataspaces for an I/O operation to build up information about the portion of each chunk needed o perform the I/O access and then iterates through the affected chunks, transferring data between memory and disk as necessary.

  Obviously, if your I/O access "goes against the grain" and requires access to many chunks, it will require more memory to store the information about the chunks affected, as well as more I/O operations to access the data in the file. If the chunk cache is large enough [possibly through the application tweaking it] and most/all of the chunk data can be held in memory, much of I/O access overhead will be amortized across multiple HDF5 dataset access operations and the against the grain access won't be so painful.

  Does that help?

    Quincey

···

On Dec 20, 2007, at 4:04 AM, Francesc Altet wrote:

My 2 cents,

--

0,0< Francesc Altet http://www.carabos.com/

V V Cárabos Coop. V. Enjoy Data
"-"

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

#2: you are rewriting over the same dataset, one chunk at a time.
read the chunk, reorder, write the chunk back into a different chunk location.
the chunk sizes I would anticipate are 64^3 or 128^3. So you would have to keep two chunks in ram;
one chunk that is being reordered, and the chunk that will be overwritten in storage.

if compression is used, then this complicates the matter.
it may be possible to not have two chunks. read a chunk, reorder, write back to same chunk; modify the pointer to chunks instead.

Matthew Dougherty
713-433-3849
National Center for Macromolecular Imaging
Baylor College of Medicine/Houston Texas USA

···

=========================================================================

Finally, I don't understand well the difference between 1) and 2), but
if the only difference is that 2) implies having two different datasets
on disk simultaneously, then 1) is clearly better, IMO.

not clear on this

1) to build up information about the portion.
what information is built?

2) how does chunk cache work? if I have a 1TB 3D image, chunks are 64^3, and I am looking at a ROI 512^3.
the ROI shifts interactively, so chunks are being accessed from storage and others being discarded from memory.
We skip the reordering problem for simplicity. Does chunk cache contain references to all the possible chunks?
Like a mini-3D dataset with pointers to all chunk storage locations?

Matthew Dougherty
713-433-3849
National Center for Macromolecular Imaging
Baylor College of Medicine/Houston Texas USA

···

=========================================================================

HDF5 currently (in the 1.6.x & 1.8.x
branches) examines the file and memory dataspaces for an I/O
operation to build up information about the portion of each chunk
needed o perform the I/O access

  Obviously, if your I/O access "goes against the grain" and requires
access to many chunks, it will require more memory to store the
information about the chunks affected, as well as more I/O operations
to access the data in the file. If the chunk cache is large enough
[possibly through the application tweaking it] and most/all of the
chunk data can be held in memory, much of I/O access overhead will be
amortized across multiple HDF5 dataset access operations and the
against the grain access won't be so painful.

A Thursday 20 December 2007, Quincey Koziol escrigué:

···

On Dec 20, 2007, at 4:04 AM, Francesc Altet wrote:
> A Thursday 20 December 2007, Dougherty, Matthew T. escrigué:
>> If I had a situation where I create a very large chunked dataset
>> that had xyz ordering, and later find out that the preferred
>> readout should be yzx; then what would the best performance
>> solution be:
>>
>> 1) recreate a new dataset with the yzx ordering
>> 2) keep the old dataset and write software to reorganize the data
>> 3) not worry about it because the performance hit is small because
>> the data can be quickly reorganized as it is coming off storage
>> into ram. 4) keep the old dataset, but coding changes deep in HDF
>> will be needed to achieve good performance.
>
> Well, I'd say that this largely depends on what kind of read
> performance
> would you expect to obtain. If, as you said, your dataset is very
> large, then the chunks should be large as well, and reading a whole
> chunk could be very costly in terms of disk I/O effort but also in
> terms of CPU if your dataset is compressed. So, if I were using
> compression and wanted to obtain maximum performance, I'd choose
> option
> 1) with no discussion, as there is very little that the software
> can do
> to avoid the necessary disk/CPU resources consumption that implies
> an 'out-of-order' read.
>
> This is in the case that you are using compression. However,
> experiments that I've carried some time ago tend to show that a
> chunk has not necessarily to be read completely as long it is not
> compressed,
> i.e. partial chunk reads are possible (Quincey, correct me if I'm
> wrong). In that case, perhaps options 3) and 4) could be feasible
> too.
>
> Finally, I don't understand well the difference between 1) and 2),
> but if the only difference is that 2) implies having two different
> datasets
> on disk simultaneously, then 1) is clearly better, IMO.

  Frankly, it could be any of the options above, depending on your
performance requirements, etc. HDF5 currently (in the 1.6.x & 1.8.x
branches) examines the file and memory dataspaces for an I/O
operation to build up information about the portion of each chunk
needed o perform the I/O access and then iterates through the
affected chunks, transferring data between memory and disk as
necessary.

  Obviously, if your I/O access "goes against the grain" and requires
access to many chunks, it will require more memory to store the
information about the chunks affected, as well as more I/O operations
to access the data in the file. If the chunk cache is large enough
[possibly through the application tweaking it] and most/all of the
chunk data can be held in memory, much of I/O access overhead will be
amortized across multiple HDF5 dataset access operations and the
against the grain access won't be so painful.

Yeah, I forgot about the fact that the chunk cache size is configurable
in forthcoming HDF5 1.8.0. That's a very interesting option indeed!
Do you know if there is any important drawback of pushing that cache
towards very large values (e.g. 1 GB)?

Cheers,

--

0,0< Francesc Altet http://www.carabos.com/

V V Cárabos Coop. V. Enjoy Data
"-"

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Hi Francesc,

A Thursday 20 December 2007, Quincey Koziol escrigué:

A Thursday 20 December 2007, Dougherty, Matthew T. escrigué:

If I had a situation where I create a very large chunked dataset
that had xyz ordering, and later find out that the preferred
readout should be yzx; then what would the best performance
solution be:

1) recreate a new dataset with the yzx ordering
2) keep the old dataset and write software to reorganize the data
3) not worry about it because the performance hit is small because
the data can be quickly reorganized as it is coming off storage
into ram. 4) keep the old dataset, but coding changes deep in HDF
will be needed to achieve good performance.

Well, I'd say that this largely depends on what kind of read
performance
would you expect to obtain. If, as you said, your dataset is very
large, then the chunks should be large as well, and reading a whole
chunk could be very costly in terms of disk I/O effort but also in
terms of CPU if your dataset is compressed. So, if I were using
compression and wanted to obtain maximum performance, I'd choose
option
1) with no discussion, as there is very little that the software
can do
to avoid the necessary disk/CPU resources consumption that implies
an 'out-of-order' read.

This is in the case that you are using compression. However,
experiments that I've carried some time ago tend to show that a
chunk has not necessarily to be read completely as long it is not
compressed,
i.e. partial chunk reads are possible (Quincey, correct me if I'm
wrong). In that case, perhaps options 3) and 4) could be feasible
too.

Finally, I don't understand well the difference between 1) and 2),
but if the only difference is that 2) implies having two different
datasets
on disk simultaneously, then 1) is clearly better, IMO.

  Frankly, it could be any of the options above, depending on your
performance requirements, etc. HDF5 currently (in the 1.6.x & 1.8.x
branches) examines the file and memory dataspaces for an I/O
operation to build up information about the portion of each chunk
needed o perform the I/O access and then iterates through the
affected chunks, transferring data between memory and disk as
necessary.

  Obviously, if your I/O access "goes against the grain" and requires
access to many chunks, it will require more memory to store the
information about the chunks affected, as well as more I/O operations
to access the data in the file. If the chunk cache is large enough
[possibly through the application tweaking it] and most/all of the
chunk data can be held in memory, much of I/O access overhead will be
amortized across multiple HDF5 dataset access operations and the
against the grain access won't be so painful.

Yeah, I forgot about the fact that the chunk cache size is configurable
in forthcoming HDF5 1.8.0. That's a very interesting option indeed!
Do you know if there is any important drawback of pushing that cache
towards very large values (e.g. 1 GB)?

  As long as the application has the memory, making the chunk cache larger doesn't have any real downside (from the HDF5 library's perspective).

  Quincey

···

On Dec 20, 2007, at 10:04 AM, Francesc Altet wrote:

On Dec 20, 2007, at 4:04 AM, Francesc Altet wrote:

Cheers,

--

0,0< Francesc Altet http://www.carabos.com/

V V Cárabos Coop. V. Enjoy Data
"-"

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Hi Matthew,

not clear on this

1) to build up information about the portion.
what information is built?

  Basically an internal representation of the elements to access on disk and in memory for each chunk.

2) how does chunk cache work? if I have a 1TB 3D image, chunks are 64^3, and I am looking at a ROI 512^3.
the ROI shifts interactively, so chunks are being accessed from storage and others being discarded from memory.
We skip the reordering problem for simplicity. Does chunk cache contain references to all the possible chunks?
Like a mini-3D dataset with pointers to all chunk storage locations?

  The chunk cache in the HDF5 library holds the actual "raw data" elements from the chunked dataset (i.e the ROI elements). The chunks are located through another data structure that indexes them, but that information is put into the "metadata cache" in the HDF5 library. In the 1.8.x library, each chunked dataset has it's own chunk cache, but the metadata cache is global to all the metadata in the file.

  Quincey

···

On Dec 20, 2007, at 12:43 PM, Dougherty, Matthew T. wrote:

Matthew Dougherty
713-433-3849
National Center for Macromolecular Imaging
Baylor College of Medicine/Houston Texas USA

=========================================================================

HDF5 currently (in the 1.6.x & 1.8.x
branches) examines the file and memory dataspaces for an I/O
operation to build up information about the portion of each chunk
needed o perform the I/O access

        Obviously, if your I/O access "goes against the grain" and requires
access to many chunks, it will require more memory to store the
information about the chunks affected, as well as more I/O operations
to access the data in the file. If the chunk cache is large enough
[possibly through the application tweaking it] and most/all of the
chunk data can be held in memory, much of I/O access overhead will be
amortized across multiple HDF5 dataset access operations and the
against the grain access won't be so painful.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

A Thursday 20 December 2007, escriguéreu:

Hi Francesc,

> A Thursday 20 December 2007, Quincey Koziol escrigué:
>>> A Thursday 20 December 2007, Dougherty, Matthew T. escrigué:
>>>> If I had a situation where I create a very large chunked dataset
>>>> that had xyz ordering, and later find out that the preferred
>>>> readout should be yzx; then what would the best performance
>>>> solution be:
>>>>
>>>> 1) recreate a new dataset with the yzx ordering
>>>> 2) keep the old dataset and write software to reorganize the
>>>> data 3) not worry about it because the performance hit is small
>>>> because the data can be quickly reorganized as it is coming off
>>>> storage into ram. 4) keep the old dataset, but coding changes
>>>> deep in HDF will be needed to achieve good performance.
>>>
>>> Well, I'd say that this largely depends on what kind of read
>>> performance
>>> would you expect to obtain. If, as you said, your dataset is
>>> very large, then the chunks should be large as well, and reading
>>> a whole chunk could be very costly in terms of disk I/O effort
>>> but also in terms of CPU if your dataset is compressed. So, if I
>>> were using compression and wanted to obtain maximum performance,
>>> I'd choose option
>>> 1) with no discussion, as there is very little that the software
>>> can do
>>> to avoid the necessary disk/CPU resources consumption that
>>> implies an 'out-of-order' read.
>>>
>>> This is in the case that you are using compression. However,
>>> experiments that I've carried some time ago tend to show that a
>>> chunk has not necessarily to be read completely as long it is not
>>> compressed,
>>> i.e. partial chunk reads are possible (Quincey, correct me if I'm
>>> wrong). In that case, perhaps options 3) and 4) could be
>>> feasible too.
>>>
>>> Finally, I don't understand well the difference between 1) and
>>> 2), but if the only difference is that 2) implies having two
>>> different datasets
>>> on disk simultaneously, then 1) is clearly better, IMO.
>>
>> Frankly, it could be any of the options above, depending on your
>> performance requirements, etc. HDF5 currently (in the 1.6.x &
>> 1.8.x branches) examines the file and memory dataspaces for an I/O
>> operation to build up information about the portion of each chunk
>> needed o perform the I/O access and then iterates through the
>> affected chunks, transferring data between memory and disk as
>> necessary.
>>
>> Obviously, if your I/O access "goes against the grain" and
>> requires access to many chunks, it will require more memory to
>> store the information about the chunks affected, as well as more
>> I/O operations to access the data in the file. If the chunk cache
>> is large enough [possibly through the application tweaking it] and
>> most/all of the chunk data can be held in memory, much of I/O
>> access overhead will be amortized across multiple HDF5 dataset
>> access operations and the against the grain access won't be so
>> painful.
>
> Yeah, I forgot about the fact that the chunk cache size is
> configurable
> in forthcoming HDF5 1.8.0. That's a very interesting option
> indeed! Do you know if there is any important drawback of pushing
> that cache towards very large values (e.g. 1 GB)?

  As long as the application has the memory, making the chunk cache
larger doesn't have any real downside (from the HDF5 library's
perspective).

Well, it is clear that HDF5 1.8 series are bringing many exciting new
features :slight_smile:

Cheers,

···

On Dec 20, 2007, at 10:04 AM, Francesc Altet wrote:
>> On Dec 20, 2007, at 4:04 AM, Francesc Altet wrote:

--

0,0< Francesc Altet http://www.carabos.com/

V V Cárabos Coop. V. Enjoy Data
"-"

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Matthew Dougherty
713-433-3849
National Center for Macromolecular Imaging
Baylor College of Medicine/Houston Texas USA

···

=========================================================================

back to the reordering question, just one dataset per hdf file for simplicity.

This relates to option #4.

If metadata cache contains all the references to the chunks in storage;
then how difficult would it be to reorder the references to chunks?

what I am looking for is an api that allows me to arbitrarily reorder the axes.
I believe there is an api that does fortran ordering(default) and another for C ordering which is reverse on the axis sequence.

seems like this would be the area of code that could be modified.

-----Original Message-----
From: Quincey Koziol [mailto:koziol@hdfgroup.org]
Sent: Thu 12/20/2007 12:56 PM
To: Dougherty, Matthew T.
Cc: hdf-forum@hdfgroup.org
Subject: Re: hypthetical question on reordering axes

  Basically an internal representation of the elements to access on
disk and in memory for each chunk.

  The chunk cache in the HDF5 library holds the actual "raw data"
elements from the chunked dataset (i.e the ROI elements). The chunks
are located through another data structure that indexes them, but
that information is put into the "metadata cache" in the HDF5
library. In the 1.8.x library, each chunked dataset has it's own
chunk cache, but the metadata cache is global to all the metadata in
the file.

Hi Matthew,

Matthew Dougherty
713-433-3849
National Center for Macromolecular Imaging
Baylor College of Medicine/Houston Texas USA

=========================================================================

back to the reordering question, just one dataset per hdf file for simplicity.

This relates to option #4.

If metadata cache contains all the references to the chunks in storage;
then how difficult would it be to reorder the references to chunks?

  This really isn't going to help, since all the data in the chunks would have to be reordered as well.

what I am looking for is an api that allows me to arbitrarily reorder the axes.
I believe there is an api that does fortran ordering(default) and another for C ordering which is reverse on the axis sequence.

  We don't currently have an API for this. We have designed the dataspace description to hold a "permutation vector" which could describe this, but it's currently not implemented.

  Quincey

···

On Dec 20, 2007, at 1:11 PM, Dougherty, Matthew T. wrote:

seems like this would be the area of code that could be modified.

-----Original Message-----
From: Quincey Koziol [mailto:koziol@hdfgroup.org]
Sent: Thu 12/20/2007 12:56 PM
To: Dougherty, Matthew T.
Cc: hdf-forum@hdfgroup.org
Subject: Re: hypthetical question on reordering axes

        Basically an internal representation of the elements to access on
disk and in memory for each chunk.

        The chunk cache in the HDF5 library holds the actual "raw data"
elements from the chunked dataset (i.e the ROI elements). The chunks
are located through another data structure that indexes them, but
that information is put into the "metadata cache" in the HDF5
library. In the 1.8.x library, each chunked dataset has it's own
chunk cache, but the metadata cache is global to all the metadata in
the file.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.