Iterate over chunk info

Hi,

I am dabbling with a thread-safe HDF5 reader in rust (https://crates.io/crates/hidefix). To do this I create my own index of chunks of each dataset. Looking into the HDF5 source code I can see that chunk information is stored in various data structures (e.g. BTreeMap) and access through chunk_info requires iterating on that structure to find a specific chunk based on e.g. number or offset. I want to access all chunks, and naturally this is super in-efficient when I have to iterate through a lot of chunks every time to find the next one. Would it be possible to expose the iterator or some other way to get a range of chunks? Not necessarily sorted. Would you be interested in a patch that does this?

Cheers, Gaute

See pull request for proposal here:https://github.com/HDFGroup/hdf5/pull/6

About 200x speedup for retrieving all chunk info for users of HDF5 for a 1.5gb file.

1 Like