# Thread-parallel compression filters? - Feature request **URL:** \ **Category:** HDF5 Library\ **Created:** [December 31, 2022, 11:35am UTC](https://forum.hdfgroup.org/t/thread-parallel-compression-filters-feature-request/10656 "2022-12-31T11:35:45Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![matthieu.schaller](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/m/9dc877/32.png) [@matthieu.schaller](https://forum.hdfgroup.org/u/matthieu.schaller)\ **Post date:** [December 31, 2022, 11:35am UTC](https://forum.hdfgroup.org/t/thread-parallel-compression-filters-feature-request/10656/1 "2022-12-31T11:35:45Z")
Hi all, I have a large production application that uses MPI + threads. The i/o pattern is either pure parallel-hdf5 or one file/rank (in a similar way to the VFD interface but done by hand). I make heavy use of compression filters. As my application is MPI + threads, only one thread per MPI rank actually does all the i/o work. In an uncompressed scenario, that is fine as I can then saturate the Infiniband cards with my 2 (or 4) MPI ranks per node. However, when compressing, this gets a lot slower. Only one thread does the compression and this now the clear bottleneck of the whole operation. It seems a bit silly as I have many idling threads which could all be drafted in for parallel compression. Would it hence be possible to see the use of `pigz` over plain `gzip` in future releases? Note that I’d like to avoid creating my own custom filter as I’d then have to distribute it collaborators using simulation results. Too high a barrier for access for many unfortunately. ---
**Author:** ![kittisopikulm](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/kittisopikulm/32/1505_2.png) [@kittisopikulm](https://forum.hdfgroup.org/u/kittisopikulm)\ **Post date:** [January 1, 2023, 10:29am UTC](https://forum.hdfgroup.org/t/thread-parallel-compression-filters-feature-request/10656/2 "2023-01-01T10:29:25Z")
There are a few applications now that have implemented thread parallel compression and decompression: > **[GitHub - imaris/ImarisWriter](https://github.com/imaris/ImarisWriter)** > > Contribute to imaris/ImarisWriter development by creating an account on GitHub. > **[Blosc2 Meets PyTables: Making HDF5 I/O Performance Awesome](https://www.blosc.org/posts/blosc2-pytables-perf/)** > > PyTables lets users to easily handle data tables and array objects in a hierarchical structure. It also supports a variety of different data compression libraries through HDF5 filters. With the relea One trick is to use [`H5Dread_chunk`](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_d.html#title30) or [`H5Dwrite_chunk`](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_d.html#title38). This will allow you to read or write the chunk directly in its compressed form. You can then setup the thread-parallel compression or decompression yourself. Another approach is using [`H5Dget_chunk_info`](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_d.html#gaccff213d3e0765b86f66d08dd9959807) to query the location of a chunk within the file. [`H5Dchunk_iter`](https://docs.hdfgroup.org/hdf5/v1_14/group___h5_d.html#title6) provides a faster way to do this, particularly if you want to get this information for all the chunks, but this is a relatively new API function. The source for many of the filters is located in the following repository. > **[GitHub - HDFGroup/hdf5\_plugins](https://github.com/HDFGroup/hdf5_plugins)** > > Contribute to HDFGroup/hdf5\_plugins development by creating an account on GitHub. For example, the code for the Zstd filter is here: > From the source code there, you can see it simply uses `ZSTD_decompress` or `ZSTD_compress`. It would be pretty easy to swap that out for `ZSTD_compressCCtx` or `ZSTD_decompressDCtx` and provide the parameter `ZSTD_c_nbWorkers` to use multiple threads per chunk. However, I suspect that having multiple threads deal with individual chunks may be more efficient. This depends on your chunking scheme. [http://facebook.github.io/zstd/zstd\_manual.html#Chapter4](http://facebook.github.io/zstd/zstd_manual.html#Chapter4) ---
**Author:** ![matthieu.schaller](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/m/9dc877/32.png) [@matthieu.schaller](https://forum.hdfgroup.org/u/matthieu.schaller)\ **Post date:** [January 5, 2023, 10:10am UTC](https://forum.hdfgroup.org/t/thread-parallel-compression-filters-feature-request/10656/3 "2023-01-05T10:10:35Z")
Thanks! Good to see that the need is not unique indeed. I’d worry though that this would lead to people using my data needing a custom filter (and hence a custom h5py) to make use of the data we produce. I’d have to somehow fudge things to make each chunk look like it was compressed with the vanilla gzip filter. Seems a bit of a hack. Also, I am exploiting some DScale filters prior to gzip so all this would need to be done by hand too, which seems like a lot of risky work to do for a library outsider. ---
**Author:** ![kittisopikulm](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/kittisopikulm/32/1505_2.png) [@kittisopikulm](https://forum.hdfgroup.org/u/kittisopikulm)\ **Post date:** [January 5, 2023, 8:33pm UTC](https://forum.hdfgroup.org/t/thread-parallel-compression-filters-feature-request/10656/4 "2023-01-05T20:33:44Z")
The question is how are people obtaining h5py. Most people either obtain h5py from pip or conda. PyPI (via pip) [https://pypi.org/project/hdf5plugin/](https://pypi.org/project/hdf5plugin/) Conda-Forge (conda) [https://anaconda.org/conda-forge/hdf5plugin](https://anaconda.org/conda-forge/hdf5plugin) The source repository for those packages is maintained by the synchrotron community via the [silx project](https://www.silx.org/). > **[GitHub - silx-kit/hdf5plugin: Set of compression filters for h5py](https://github.com/silx-kit/hdf5plugin)** > > Set of compression filters for h5py Additionally, zstandard is an open source project maintained by Meta (fma Facebook): [http://facebook.github.io/zstd/](http://facebook.github.io/zstd/) These third party filters are generally open, free, and registered: [https://portal.hdfgroup.org/display/support/Registered+Filter+Plugins](https://portal.hdfgroup.org/display/support/Registered+Filter+Plugins) The HDF Group maintains a repository that collects the source code of these filters: > **[GitHub - HDFGroup/hdf5\_plugins](https://github.com/HDFGroup/hdf5_plugins)** > > Contribute to HDFGroup/hdf5\_plugins development by creating an account on GitHub. There is an ongoing conversation along these lines in another thread: > [@What do you want to see in "HDF5 2.0"?](https://forum.hdfgroup.org/t/what-do-you-want-to-see-in-hdf5-2-0/10003/50): > > When it comes to filters, we’ll be working to better integrate the filter plugins rather than creating built-in filters.