Hi,
I’m trying to improve JLD2 a julia library for storing files based on HDF5. It implements a subset of HDF5.
I want to implement chunking using Fixed Array Indexing
so that I can utilize Blosc compression - since Blosc can only compress data chunks < 2GB so I need to automatically break up the array into chunks.
(If this is not the correct thing to do then I’m open for suggestions)
I have tried to implement everything as stated in the Documentation but I have a few questions that I couldn’t figure out using just the docs:
-
In
Chunked Storage Property Description: What is the meaning of thechunked layout feature flag? -
How is
pagingin the Fixed Array Data Block turned on and off. The docs state two versions (with paging/without paging) but I don’t know how to specify either. -
Similarly what happens to the
page bitsfields when paging is turned off? -
In
Data Block Element for Filtered Dataset Chunk: What is the meaning of theFilter Mask? -
How are boundary (incomplete) chunks handled / specified?
Kind regards,
Jonas