# Efficient access of sparse, chunked datasets

**URL:** <https://forum.hdfgroup.org/t/efficient-access-of-sparse-chunked-datasets/10969>\
**Category:** h5py\
**Created:** [March 29, 2023, 2:55am UTC](https://forum.hdfgroup.org/t/efficient-access-of-sparse-chunked-datasets/10969 "2023-03-29T02:55:41Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rtkeskitalo](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/94ad74/32.png) [@rtkeskitalo](https://forum.hdfgroup.org/u/rtkeskitalo)\
**Post date:** [March 29, 2023, 2:55am UTC](https://forum.hdfgroup.org/t/efficient-access-of-sparse-chunked-datasets/10969/1 "2023-03-29T02:55:41Z")

</div>

I have a sparse dataset in chunked storage written to HDF5 using h5py. Only the chunks that have data are written to disk so there must be some metadata that identifies the non-empty chunks.

I want to open the file and operate on the non-empty chunks. However, dset.iter\_chunks() will loop over all chunks. There must be a list of non-empty chunks _somewhere_. How do I access it?

---

<div class="post-metadata">

**Author:** ![thomas1](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/thomas1/32/1733_2.png) [@thomas1](https://forum.hdfgroup.org/u/thomas1)\
**Post date:** [March 30, 2023, 3:39pm UTC](https://forum.hdfgroup.org/t/efficient-access-of-sparse-chunked-datasets/10969/2 "2023-03-30T15:39:26Z")

</div>

Hi @rtkeskitalo - getting the allocated chunk details is part of h5py’s [low-level API](https://api.h5py.org/h5d.html#dataset-objects). You [get to the low level](https://docs.h5py.org/en/stable/high/lowlevel.html) via a `.id` attribute on the dataset object. So you can do something like this:

```python
[ds.id.get_chunk_info(i) for i in range(ds.id.get_num_chunks())]

```

Each chunk info object has a `chunk_offset` attribute which tells you where in the dataset that chunk starts.

---

<div class="post-metadata">

**Author:** ![rtkeskitalo](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/94ad74/32.png) [@rtkeskitalo](https://forum.hdfgroup.org/u/rtkeskitalo)\
**Post date:** [March 30, 2023, 9:22pm UTC](https://forum.hdfgroup.org/t/efficient-access-of-sparse-chunked-datasets/10969/3 "2023-03-30T21:22:28Z")

</div>

Thanks! That was exactly what I was looking for.

Iterating over the non-empty chunks this way is efficient but a little clunky. I wonder if `dset.iter_chunks()` should have a keyword argument to skip empty chunks or if there should be another method to iterate over them?

---

<div class="post-metadata">

**Author:** ![thomas1](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/thomas1/32/1733_2.png) [@thomas1](https://forum.hdfgroup.org/u/thomas1)\
**Post date:** [April 6, 2023, 5:07pm UTC](https://forum.hdfgroup.org/t/efficient-access-of-sparse-chunked-datasets/10969/4 "2023-04-06T17:07:53Z")

</div>

That sounds reasonable. I’ve [opened an issue](https://github.com/h5py/h5py/issues/2249) about it, and you’re welcome to make a PR. h5py has [contributor docs](https://docs.h5py.org/en/stable/contributing.html#how-to-get-your-code-into-h5py) that might help.

Note that _unallocated_ isn’t precisely the same as _empty_. You can have allocated but empty chunks if you write empty data to them, or if you [change the allocation time](https://confluence.hdfgroup.org/display/HDF5/H5P_SET_ALLOC_TIME). And the dataset can have a fill value, so unallocated space might logically be full of -1 or whatever you’ve set the fill value to, rather than zeros. In a lot of scenarios this distinction won’t matter, of course, but if it’s an option in h5py we’ll need to ensure it’s clearly described. 🙂

---

<div class="post-metadata">

**Author:** ![kittisopikulm](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/kittisopikulm/32/1505_2.png) [@kittisopikulm](https://forum.hdfgroup.org/u/kittisopikulm)\
**Post date:** [May 3, 2023, 2:09am UTC](https://forum.hdfgroup.org/t/efficient-access-of-sparse-chunked-datasets/10969/5 "2023-05-03T02:09:18Z")

</div>

I provided an implementation of a method to iterate over only allocated chunks here:

> <https://github.com/h5py/h5py/issues/2249#issuecomment-1532366113>

The C function [`H5Dchunk_iter`](https://docs.hdfgroup.org/hdf5/develop/group___h5_d.html#title6) only iterates over allocated chunks.

Here is the method `get_allocated_chunk_info`:

```auto
def get_allocated_chunk_info(dataset):
     chunk_info = []
     dataset.id.chunk_iter(lambda v: chunk_info.append(v))
     return chunk_info

```

Here is how to use it.

```auto
In [93]: with h5py.File("pytest.h5", "w") as f:
    ...: dset = f.create_dataset("test", (1024*4,1024*4), chunks=(16,16))
    ...: dset[0,0] = 1
    ...: dset[16,16] = 1
    ...: print(dset.id.get_num_chunks())
    ...: 
2

In [94]: def get_allocated_chunk_info(dataset):
    ...: chunk_info = []
    ...: dataset.id.chunk_iter(lambda v: chunk_info.append(v))
    ...: return chunk_info
    ...: 

In [95]: with h5py.File("pytest.h5", "r") as f:
    ...: out = get_allocated_chunk_info(f["test"])
    ...: 

In [96]: out
Out[96]: 
[StoreInfo(chunk_offset=(0, 0), filter_mask=0, byte_offset=4016, size=1024),
 StoreInfo(chunk_offset=(16, 16), filter_mask=0, byte_offset=5040, size=1024)]

```

This method is highly efficiently allowing you to query 65,536 chunks in 0.272 seconds and only requiring less than 9 bytes per chunk.

```auto
In [100]: %%time
     ...: with h5py.File("pytest.h5", "w") as f:
     ...: dset = f.create_dataset("test", (1024*4,1024*4), chunks=(16,16))
     ...: dset[:] = 1
     ...: print(dset.id.get_num_chunks())
     ...: 
65536
CPU times: user 4.03 s, sys: 399 ms, total: 4.43 s
Wall time: 4.48 s

In [101]: %%time
     ...: with h5py.File("pytest.h5", "r") as f:
     ...: print(f["test"].id.get_num_chunks())
     ...: out = get_allocated_chunk_info(f["test"])
     ...: 
65536
CPU times: user 255 ms, sys: 20 ms, total: 275 ms
Wall time: 272 ms

In [103]: sys.getsizeof(out) / 1024
Out[103]: 549.3046875

```
