Implementing Chunking - Fixed Array Index


I’m trying to improve JLD2 a julia library for storing files based on HDF5. It implements a subset of HDF5.

I want to implement chunking using Fixed Array Indexing
so that I can utilize Blosc compression - since Blosc can only compress data chunks < 2GB so I need to automatically break up the array into chunks.
(If this is not the correct thing to do then I’m open for suggestions)

I have tried to implement everything as stated in the Documentation but I have a few questions that I couldn’t figure out using just the docs:

  • In Chunked Storage Property Description: What is the meaning of the chunked layout feature flag?

  • How is paging in the Fixed Array Data Block turned on and off. The docs state two versions (with paging/without paging) but I don’t know how to specify either.

  • Similarly what happens to the page bits fields when paging is turned off?

  • In Data Block Element for Filtered Dataset Chunk: What is the meaning of the Filter Mask ?

  • How are boundary (incomplete) chunks handled / specified?

Kind regards,