Read All Available Chunk in a Dataset.

Hdf-forum archives
1

I have a dataset of size 16*16
I write Chunk of Data of size 4*4.

Some data aren't available so i wrote only 3 chunk on my dataset.

How to read only the 3 available chunk?

Thanks

Mathieu

2

Mathieu, don't worry about the chunks. Just read/write what you need
in terms of the dataspace (hyperslab, point selection...) and the library
will determine for you which chunks are touched (read/written) by the
operation.
It'll just return fill values for un-allocated/partially filled chunks.

Best, G.

···

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Mathieu westphal
Sent: Thursday, May 10, 2012 6:46 AM
To: hdf-forum@hdfgroup.org
Subject: [Hdf-forum] Read All Available Chunk in a Dataset.

I have a dataset of size 16*16
I write Chunk of Data of size 4*4.

Some data aren't available so i wrote only 3 chunk on my dataset.

How to read only the 3 available chunk?

Thanks

Mathieu

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

3

Mathieu, don't worry about the chunks. Just read/write what you need
in terms of the dataspace (hyperslab, point selection...) and the library
will determine for you which chunks are touched (read/written) by the
operation.
It'll just return fill values for un-allocated/partially filled chunks.

Yep you're right! But the time to read the data is linear with the size of all the Dataset, not only of the size of available data !
My real dataset is really sparse ! i need to read all available value without taking time to read unavailable value.
Is there anyway to select only real value?

Anyway how to know how many value are really read from dataset?

Thanks

Mathieu

···

On 05/10/2012 02:01 PM, Gerd Heber wrote:

Best, G.

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Mathieu westphal
Sent: Thursday, May 10, 2012 6:46 AM
To: hdf-forum@hdfgroup.org
Subject: [Hdf-forum] Read All Available Chunk in a Dataset.

I have a dataset of size 16*16
I write Chunk of Data of size 4*4.

Some data aren't available so i wrote only 3 chunk on my dataset.

How to read only the 3 available chunk?

Thanks

Mathieu

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

4

That's why I asked you to define the smallest hyperslab or union
of hyperslabs, or point selections that cover you chunks and
read just those. The library will locate just those chunks
(tree traversal, not linear) and deal with them.
Have a look at section 7.4. Dataspaces and Data Transfer in the
user guide!

Best, G.

···

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Mathieu westphal
Sent: Thursday, May 10, 2012 8:15 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Read All Available Chunk in a Dataset.

On 05/10/2012 02:01 PM, Gerd Heber wrote:

Mathieu, don't worry about the chunks. Just read/write what you need
in terms of the dataspace (hyperslab, point selection...) and the
library will determine for you which chunks are touched (read/written)
by the operation.
It'll just return fill values for un-allocated/partially filled chunks.

Yep you're right! But the time to read the data is linear with the size of
all the Dataset, not only of the size of available data !
My real dataset is really sparse ! i need to read all available value
without taking time to read unavailable value.
Is there anyway to select only real value?

Anyway how to know how many value are really read from dataset?

Thanks

Mathieu

Best, G.

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org
[mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Mathieu westphal
Sent: Thursday, May 10, 2012 6:46 AM
To: hdf-forum@hdfgroup.org
Subject: [Hdf-forum] Read All Available Chunk in a Dataset.

I have a dataset of size 16*16
I write Chunk of Data of size 4*4.

Some data aren't available so i wrote only 3 chunk on my dataset.

How to read only the 3 available chunk?

Thanks

Mathieu

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

5

Thank you ! So as i understand there is no way to automatically select availlable data without knowing where it is.

WHen i read the data, i normally dont know wich part of the dataset is full of data and wich part is not, so i will have to store an hyperslab wich describe where the available data is.

Thanks for your help

Mathieu

Quoting Gerd Heber <gheber@hdfgroup.org>:

···

That's why I asked you to define the smallest hyperslab or union
of hyperslabs, or point selections that cover you chunks and
read just those. The library will locate just those chunks
(tree traversal, not linear) and deal with them.
Have a look at section 7.4. Dataspaces and Data Transfer in the
user guide!

Best, G.

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Mathieu westphal
Sent: Thursday, May 10, 2012 8:15 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Read All Available Chunk in a Dataset.

On 05/10/2012 02:01 PM, Gerd Heber wrote:

Mathieu, don't worry about the chunks. Just read/write what you need
in terms of the dataspace (hyperslab, point selection...) and the
library will determine for you which chunks are touched (read/written)
by the operation.
It'll just return fill values for un-allocated/partially filled chunks.

Yep you're right! But the time to read the data is linear with the size of
all the Dataset, not only of the size of available data !
My real dataset is really sparse ! i need to read all available value
without taking time to read unavailable value.
Is there anyway to select only real value?

Anyway how to know how many value are really read from dataset?

Thanks

Mathieu

Best, G.

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org
[mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Mathieu westphal
Sent: Thursday, May 10, 2012 6:46 AM
To: hdf-forum@hdfgroup.org
Subject: [Hdf-forum] Read All Available Chunk in a Dataset.

I have a dataset of size 16*16
I write Chunk of Data of size 4*4.

Some data aren't available so i wrote only 3 chunk on my dataset.

How to read only the 3 available chunk?

Thanks

Mathieu

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

6

Mathieu, (simple) HDF5 datasets are logically dense arrays.
With chunking you can mimic sparse storage, but you loose
track of where the non-zero entries are.
If you need that kind of information, you have to keep track
of that with, for example, a region reference or emulate
Compressed Row Storage or other sparse format directly
with more than one dataset.

Best, G.

···

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of mathieu.westphal@obs.ujf-grenoble.fr
Sent: Friday, May 11, 2012 5:24 AM
To: hdf-forum@hdfgroup.org
Subject: Re: [Hdf-forum] Read All Available Chunk in a Dataset.

Thank you ! So as i understand there is no way to automatically select
availlable data without knowing where it is.

WHen i read the data, i normally dont know wich part of the dataset is full
of data and wich part is not, so i will have to store an hyperslab wich
describe where the available data is.

Thanks for your help

Mathieu

Quoting Gerd Heber <gheber@hdfgroup.org>:

That's why I asked you to define the smallest hyperslab or union of
hyperslabs, or point selections that cover you chunks and read just
those. The library will locate just those chunks (tree traversal, not
linear) and deal with them.
Have a look at section 7.4. Dataspaces and Data Transfer in the user
guide!

Best, G.

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org
[mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Mathieu westphal
Sent: Thursday, May 10, 2012 8:15 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Read All Available Chunk in a Dataset.

On 05/10/2012 02:01 PM, Gerd Heber wrote:

Mathieu, don't worry about the chunks. Just read/write what you need
in terms of the dataspace (hyperslab, point selection...) and the
library will determine for you which chunks are touched
(read/written) by the operation.
It'll just return fill values for un-allocated/partially filled chunks.

Yep you're right! But the time to read the data is linear with the
size of all the Dataset, not only of the size of available data !
My real dataset is really sparse ! i need to read all available value
without taking time to read unavailable value.
Is there anyway to select only real value?

Anyway how to know how many value are really read from dataset?

Thanks

Mathieu

Best, G.

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org
[mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Mathieu westphal
Sent: Thursday, May 10, 2012 6:46 AM
To: hdf-forum@hdfgroup.org
Subject: [Hdf-forum] Read All Available Chunk in a Dataset.

I have a dataset of size 16*16
I write Chunk of Data of size 4*4.

Some data aren't available so i wrote only 3 chunk on my dataset.

How to read only the 3 available chunk?

Thanks

Mathieu

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org