Experience with 'FPC' compressor in concert with HDF5

Hdf-forum archives
1

I searched the HDF5-Forum archives and didn't get any hits on 'FPC'

I am curious if anyone has any experience adapting the 'FPC' floating
point compression code,
http://www.csl.cornell.edu/~burtscher/research/FPC/
for use as an internal HDF5 compression 'filter'?

If so, what where your experiences? Would you be willing to share your
code? Do you know of anything 'better' for floating point data?

Thanks for any info.

Mark

···

--
Mark C. Miller, Lawrence Livermore National Laboratory
<<<<<<<<<<<<<<<<!!LLNL BUSINESS ONLY!!>>>>>>>>>>>>>>>>
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

2

In case anyone winds up following 'FPC' to this archive, I thought I
would just mention experience from a simple test I did with FPC (apart
from HDF5 though).

I created three binary files of 1,000,000 samples of 4 cycles of a sin
wave. One with a magnitude of 1.0, one with a magnitude of 1e+15 and one
with magnitude of 1e-15. So, its very smooth data!

Test FPC compression ratio gzip --best ratio
1.0 mag 2.00 1.27
1e-5 mag 2.00 1.27
1e+5 mag 2.04 1.28

So, FPC does better than gzip for this relatively smooth data and FPC
appears to compress and decompress *much* faster (5-10x) than gzip too.

Finally, I tried an integer example with a magnitude of 100000 (fitting
well within the range of a 32-bit int) and compressed that with gzip and
get a compression ratio of about 3.25.

···

On Fri, 2012-07-13 at 10:09 -0700, Mark Miller wrote:

I searched the HDF5-Forum archives and didn't get any hits on 'FPC'

I am curious if anyone has any experience adapting the 'FPC' floating
point compression code,
http://www.csl.cornell.edu/~burtscher/research/FPC/
for use as an internal HDF5 compression 'filter'?

If so, what where your experiences? Would you be willing to share your
code? Do you know of anything 'better' for floating point data?

Thanks for any info.

Mark

--
Mark C. Miller, Lawrence Livermore National Laboratory
<<<<<<<<<<<<<<<<!!LLNL BUSINESS ONLY!!>>>>>>>>>>>>>>>>
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511