Zstandard plugin for HDF5 does not allow negative compression levels


#1

Zstandard has allowed negative compressions since 2018:

@paramon , are you able to accept this pull request that I submitted in March 2022?

If @paramon is not able to respond, how should we proceed with updating the Zstandard HDF5 plugin? I notice that he has not had any Github activity since May 2019.


#2

Thanks @kittisopikulm for bringing this up. Since we are not talking about a giant piece of code, perhaps we (The HDF Group) can adopt a fork, and move forward from there. Let me talk to my colleagues!

G.


#3

That would work.

Perhaps it can go into github.com/hdfgroup/hdf5_plugins ?

Alternatively, I’ve started to prepare a fork here:

Also, welcome back!


#4

Two issues here - if I understand correctly.

  1. You have a fork that is different then the current zstandard. We do use forks of some compression libs so that we can build from source with cmake. We could create a compression repo in github for “HDF” alterations. I have always noted my changes in a README.HDF file.

  2. Add a HDF5 filter plugin to the hdf5_plugins project. That is easily doable if the filter plugin file exists and I can just use a tar.gz file (or git repo) for the compression library ExternalProject build.


#5

I am not forking Zstandard library or proposing to do so. That repository lives at https://github.com/facebook/zstd. In our testing, the compression is generally applicable and tunable to achieve high speeds and/or high compression ratio. Additionally, the implementation of the Zstandard-HDF5 plugin is compatible with the implementation in Zarr/numcodecs. Thus we are very interested in the capabilities of this compression codec.

Currently, the registered Zstandard plugin for HDF5 with filter ID 32015 is assigned to Andrey Paramonov(@paramon). I am proposing to change this registration.

Unfortunately, we have not heard from Andrey Paramonov for three years. He also has no Github activity for three years.

I am trying to update the HDF5 plugin for Zstandard to take advantage of recent capabilities of Zstandard C library from Meta (formerly Facebook).

My proposed fork is currently at the same commit as the registered upstream repository.

Six months ago I proposed a pull request to allow for negative compression levels:

There are other features of the upstream library from Meta such as dictionary compression that would be great to enable.

In summary, the original registrant of the Zstandard plugin for HDF5 appears to no longer be available with no activity for more than three years. The upstream Zstandard library from Meta has gained new capabilities since then. In order to update this plugin, I propose to change the registration.

If HDF Group would like to adopt this plugin directly or incorporate it into the main code base, that would be amazing. I would then send my pull requests there.


#6

The Zstandard is 32015, other then that, then I understand you want the to change the registered repo to the new location.

@gheber, What is the procedure to get this done?