The bitshuffle filter is now supported in HSDS. Bitshuffle works much like the standard HDF5 library byteshuffle, except that it works at the bit level. Also, bitshuffle includes it’s own lz4 compression. See: bitshuffle · PyPI for more information about bitshuffle.
For an example of how this works using the HDF REST API, see: https://github.com/HDFGroup/hsds/blob/master/tests/integ/filter_test.py#L221.
With h5pyd v0.17.0,
hsload will import any custom filters defined on a HDF5 dataset to the filter pipeline for the HSDS dataset.
If the filter is supported in HSDS (e.g. bitshuffle, blosclz, lz4, lz4hc, zlib, zstd as of HSDS v 0.8.5), the filter will be invoked on reads and writes to the dataset, otherwise it will be ignored.
Note that with HSDS, all the filters are built in (there’s no equivalent to the library filter plugin mechanism), on the other hand, HSDS supports a broader set of filters than those that come in the standard HDF5 library release. For the HSDS clients, they don’t need to be concerned with filters at all, since all compression/decompression is handled server side.
Are there other filters you’d like to see supported in HSDS? Let us know!