Adding LZ4HC and Fast-Encoder Acceleration to the LZ4 Filter Plugin (ID 32004)

Proposals
1

Hi everyone!

We’ve put together a design proposal for extending the HDF5 LZ4 filter plugin (ID 32004), and we’d love to get some community eyes on it before we move forward.

Right now, the plugin only exposes the default fast LZ4 encoder. The proposal adds two new capabilities:

  • LZ4HC — the high-compression encoder, which typically squeezes out 15–30% better compression ratios with no change to decompression speed (same bitstream, same decoder).

  • Acceleration parameter — for those on the opposite end who want to trade a little ratio for even faster write throughput.

The good news: this is fully backward-compatible. Existing files and existing reader deployments are unaffected, and no new filter ID is needed. It’s purely additive.

The full write-up — including the proposed cd_values[] layout and compatibility rules — is over on GitHub. If this is something you’d use, are already using the 32004 plugin, or if you have concerns about the design, we’d really appreciate your input!

:backhand_index_pointing_right: Please leave all comments and feedback directly on the GitHub issue: https://github.com/HDFGroup/hdf5_plugins/issues/244

Thanks in advance — looking forward to hearing what you all think!