zlib-ng recently had a stable release 2.0.2. zlib-ng standalone benchmarks indicate up to multiple times faster than zlib on write and 10’s of percent faster on read. I haven’t benchmarked with HDF5, but I noticed it was simple to build HDF5 with zlib-ng.
zlib-ng may be of interest to HDF5 community as original zlib has last release in 2017, and zlib-ng incorporates improvements from some popular zlib forks since then. I simply added CMake option to Zlib: -DZLIB_COMPAT:BOOL=on to work with HDF5.
2 Likes
@michael1 Very interesting. If you happen to benchmark HDF5 (or netCDF), please reply to this thread. I might set up my own model to see how it reacts to zlib-ng…
I have done some quick serial comparisons between “normal” zlib and zlib-ng and I get anywhere from 50% faster to 80% faster (1/2 to 1/5 the runtime) with zlib-ng. Again, not rigourous in anyway just some quick tests on some medium to very large files.
Thanks, this is interesting.
I spotted on one of the issues someone saying that “zlib-ng is optimized for larger files”. Does that imply that there’s any kind of trade-off with performance on smaller files? And if there could be, what sort of file sizes are we talking about as larger/smaller?
Even though HDF5 files are often big (GB), compression typically works on much smaller chunks - the chunk size is up to the user, but I’d guess a few KB to a few MB is common. So I’m wondering how much the possible performance benefits depend on chunk size.
