I’m trying to improve JLD2 a julia library for storing files based on HDF5. It implements a subset of HDF5.
I want to implement chunking using
Fixed Array Indexing
so that I can utilize
Blosc compression - since Blosc can only compress data chunks < 2GB so I need to automatically break up the array into chunks.
(If this is not the correct thing to do then I’m open for suggestions)
I have tried to implement everything as stated in the Documentation but I have a few questions that I couldn’t figure out using just the docs:
Chunked Storage Property Description: What is the meaning of the
chunked layout feature flag?
pagingin the Fixed Array Data Block turned on and off. The docs state two versions (with paging/without paging) but I don’t know how to specify either.
Similarly what happens to the
page bitsfields when paging is turned off?
Data Block Element for Filtered Dataset Chunk: What is the meaning of the
How are boundary (incomplete) chunks handled / specified?