New HDF5 compression plugin

miller86 · October 27, 2016, 11:53pm

Hi All,

Just wanted to mention a new HDF5 floating point compression plugin available on github...

This plugin will come embedded in the next release of the Silo library as well.

···

--
Mark C. Miller, LLNL

nelson99 · October 28, 2016, 12:53am

Nice. Thanks!

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Thursday, October 27, 2016 4:53 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] New HDF5 compression plugin

Hi All,

Just wanted to mention a new HDF5 floating point compression plugin available on github...

This plugin will come embedded in the next release of the Silo library as well.

--
Mark C. Miller, LLNL

elvis.stansvik · October 28, 2016, 11:12am

Hi All,

Just wanted to mention a new HDF5 floating point compression plugin
available on github...

https://github.com/LLNL/H5Z-ZFP

This plugin will come embedded in the next release of the Silo library as
well.

Thanks for the pointer. That's very interesting. I had not heard about
ZFP before. The ability to set a bound on the error in the lossless
case seems very useful.

Do you know if there has been any comparative benchmarks of ZFP
against other compressors?

After some basic benchmarking, we recently settled on Blosc_LZ4HC at
level 4 for our datasets (3D float tomography data), but maybe it
would be worthwhile to look at ZFP as well..

Best regards,
Elvis

···

2016-10-28 1:53 GMT+02:00 Miller, Mark C. <miller86@llnl.gov>:

--
Mark C. Miller, LLNL

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

paramon · October 28, 2016, 12:11pm

28.10.2016 2:53, Miller, Mark C. пишет:

Hi All,

Just wanted to mention a new HDF5 floating point compression plugin
available on github...

https://github.com/LLNL/H5Z-ZFP

This plugin will come embedded in the next release of the Silo library
as well.

Hello Mark!

I've downloaded library from

and even managed to compile it under Windows.
I'd like now to compare it on my data against other options (e.g. http://freearc.org/) but I couldn't grok how to specify lossless mode for zfp. What are the parameters?

Best wishes,
Andrey Paramonov

···

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

elvis.stansvik · October 31, 2016, 3:31pm

Hi All,

Just wanted to mention a new HDF5 floating point compression plugin
available on github...

https://github.com/LLNL/H5Z-ZFP

This plugin will come embedded in the next release of the Silo library as
well.

I've successfully tried using the plugin through h5py. I did have a
problem with the fixed-accuracy mode however and filed an issue:

https://github.com/LLNL/H5Z-ZFP/issues/1

It's very likely that I'm doing something wrong though.

Elvis

···

2016-10-28 1:53 GMT+02:00 Miller, Mark C. <miller86@llnl.gov>:

--
Mark C. Miller, LLNL

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

psteinb · October 28, 2016, 11:23am

I second this request big time and would add zstd, if we are already trying out various encoders.

P

···

On 10/28/2016 01:12 PM, Elvis Stansvik wrote:

2016-10-28 1:53 GMT+02:00 Miller, Mark C. <miller86@llnl.gov>:

Hi All,

Just wanted to mention a new HDF5 floating point compression plugin
available on github...

https://github.com/LLNL/H5Z-ZFP

This plugin will come embedded in the next release of the Silo library as
well.

Thanks for the pointer. That's very interesting. I had not heard about
ZFP before. The ability to set a bound on the error in the lossless
case seems very useful.

Do you know if there has been any comparative benchmarks of ZFP
against other compressors?

After some basic benchmarking, we recently settled on Blosc_LZ4HC at
level 4 for our datasets (3D float tomography data), but maybe it
would be worthwhile to look at ZFP as well..

Best regards,
Elvis

--
Mark C. Miller, LLNL

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

elvis.stansvik · October 28, 2016, 11:38am

Hi All,

Just wanted to mention a new HDF5 floating point compression plugin
available on github...

https://github.com/LLNL/H5Z-ZFP

This plugin will come embedded in the next release of the Silo library as
well.

Thanks for the pointer. That's very interesting. I had not heard about
ZFP before. The ability to set a bound on the error in the lossless
case seems very useful.

Here I meant the lossy case of course..

Elvis

···

2016-10-28 13:12 GMT+02:00 Elvis Stansvik <elvis.stansvik@orexplore.com>:

2016-10-28 1:53 GMT+02:00 Miller, Mark C. <miller86@llnl.gov>:

Do you know if there has been any comparative benchmarks of ZFP
against other compressors?

After some basic benchmarking, we recently settled on Blosc_LZ4HC at
level 4 for our datasets (3D float tomography data), but maybe it
would be worthwhile to look at ZFP as well..

Best regards,
Elvis

--
Mark C. Miller, LLNL

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

miller86 · October 28, 2016, 3:07pm

Do you know if there has been any comparative benchmarks of ZFP
against other compressors?

Yes, see here...

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of Elvis Stansvik <elvis.stansvik@orexplore.com<mailto:elvis.stansvik@orexplore.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Friday, October 28, 2016 at 4:12 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] New HDF5 compression plugin

miller86 · October 28, 2016, 3:11pm

I've downloaded library from

and even managed to compile it under Windows.
I'd like now to compare it on my data against other options (e.g.
http://freearc.org/) but I couldn't grok how to specify lossless mode
for zfp. What are the parameters?

Have a look at test_write.c test example. It demonstrates all 4 modes.

The ZFP library's compression controls are described in the ZFP release notes, here...

github.com

LLNL/zfp/blob/master/README.md

ZFP
===

INTRODUCTION
------------

zfp is an open source C/C++ library for compressed numerical arrays that
support high throughput read and write random access.  zfp also supports
streaming compression of integer and floating-point data, e.g., for
applications that read and write large data sets to and from disk.

zfp was developed at Lawrence Livermore National Laboratory and is loosely
based on the algorithm described in the following paper:

    Peter Lindstrom
    "Fixed-Rate Compressed Floating-Point Arrays"
    IEEE Transactions on Visualization and Computer Graphics
    20(12):2674-2683, December 2014
    doi:10.1109/TVCG.2014.2346458

This file has been truncated. show original

The filter's default behavior, if you specify nelmts==0, is best quality (least loss).

If you still have questions about using it after reviewing the above refs, please let me know.

Mark

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of Андрей Парамонов <paramon@acdlabs.ru<mailto:paramon@acdlabs.ru>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Friday, October 28, 2016 at 5:11 AM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] New HDF5 compression plugin

elvis.stansvik · October 28, 2016, 2:27pm

28.10.2016 2:53, Miller, Mark C. пишет:

Hi All,

Just wanted to mention a new HDF5 floating point compression plugin
available on github...

https://github.com/LLNL/H5Z-ZFP

This plugin will come embedded in the next release of the Silo library
as well.

Hello Mark!

I've downloaded library from
https://github.com/LLNL/zfp
and even managed to compile it under Windows.
I'd like now to compare it on my data against other options (e.g.
http://freearc.org/) but I couldn't grok how to specify lossless mode for
zfp. What are the parameters?

From what I understand, zfp is always lossy (but the error can be

bounded in various ways). fpzip seems to be the lossless variant, but
this filter plugin is for zfp.

The parameters for the zfp filter plugin seems to be documented in
README_MORE in the GitHub repo:

https://github.com/LLNL/H5Z-ZFP/blob/master/README_MORE

Elvis

···

2016-10-28 14:11 GMT+02:00 Андрей Парамонов <paramon@acdlabs.ru>:

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

miller86 · October 31, 2016, 4:39pm

Hi Elvis,

I've successfully tried using the plugin through h5py. I did have a
problem with the fixed-accuracy mode however and filed an issue:

https://github.com/LLNL/H5Z-ZFP/issues/1

It's very likely that I'm doing something wrong though.

Thanks so much!!

I will take a look at the issue you reported later this week.

I may have to rope the researcher who developed ZFP in to help.

AFAICT, I am calling ZFP with correct parameters and, your email confirms you see zfp_stream_set_accuracy() being called.

Mark

elvis.stansvik · October 28, 2016, 11:59am

I second this request big time and would add zstd, if we are already trying
out various encoders.

This may not be of interest, and does not include zstd, but I'm
attaching an excerpt from some of the results I got when back when
doing our basic benchmarking of some algorithms (all lossless).

It was based on those that we settled on Blosc_LZ4HC at level 4, since
we were looking for very fast decompression times, while longer
compression times and slightly larger file size was acceptable up to
certain points. The gzip results are included mostly because that's
what we were using at the time and I wanted them as a comparison, but
we knew we wanted something else. The input for those benchmarks was a
500x300x300 float dataset containing a tomographic 3D image.

I might try to dig up the script I used for the benchmark and see if
we still have the input I used, and do a test with lossy ZFP. It could
be very interesting for creating 3D "thumbnails" in our application.

Elvis

···

2016-10-28 13:23 GMT+02:00 Peter Steinbach <steinbach@scionics.de>:

P

On 10/28/2016 01:12 PM, Elvis Stansvik wrote:

2016-10-28 1:53 GMT+02:00 Miller, Mark C. <miller86@llnl.gov>:

Hi All,

Just wanted to mention a new HDF5 floating point compression plugin
available on github...

GitHub - LLNL/H5Z-ZFP: A registered ZFP compression plugin for HDF5

This plugin will come embedded in the next release of the Silo library as
well.

Thanks for the pointer. That's very interesting. I had not heard about
ZFP before. The ability to set a bound on the error in the lossless
case seems very useful.

Do you know if there has been any comparative benchmarks of ZFP
against other compressors?

After some basic benchmarking, we recently settled on Blosc_LZ4HC at
level 4 for our datasets (3D float tomography data), but maybe it
would be worthwhile to look at ZFP as well..

Best regards,
Elvis

--
Mark C. Miller, LLNL

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

elvis.stansvik · October 28, 2016, 3:22pm

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Elvis
Stansvik <elvis.stansvik@orexplore.com>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Date: Friday, October 28, 2016 at 4:12 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] New HDF5 compression plugin

Do you know if there has been any comparative benchmarks of ZFP
against other compressors?

Yes, see here...

http://computation.llnl.gov/projects/floating-point-compression/zfp-compression-ratio-and-quality

Thanks! Should have found that one myself.

Elvis

···

2016-10-28 17:07 GMT+02:00 Miller, Mark C. <miller86@llnl.gov>:

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

elvis.stansvik · October 31, 2016, 4:55pm

Hi Elvis,

I've successfully tried using the plugin through h5py. I did have a
problem with the fixed-accuracy mode however and filed an issue:

https://github.com/LLNL/H5Z-ZFP/issues/1

It's very likely that I'm doing something wrong though.

Thanks so much!!

I will take a look at the issue you reported later this week.

I may have to rope the researcher who developed ZFP in to help.

AFAICT, I am calling ZFP with correct parameters and, your email confirms

you see zfp_stream_set_accuracy() being called.

Alright, no hurry.

I've been in contact with Peter earlier with some other questions. He seems
very helpful.

And yes, zfp_stream_set_accuracy seems to be called correctly with the
parameters I pass through h5py (same as when I do the equivalent with the
zfp command line tool), so I think something else is going on. The dataset
type class and dimensions are also correctly identified by the filter
plugin.

Elvis

···

Den 31 okt. 2016 5:41 em skrev "Miller, Mark C." <miller86@llnl.gov>:

Mark

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

psteinb · October 28, 2016, 12:17pm

Hi Elvis,

interesting I am mostly looking into 3D optical tomography images (which exclusively use voxels represented by integers).

This may not be of interest, and does not include zstd, but I'm
attaching an excerpt from some of the results I got when back when
doing our basic benchmarking of some algorithms (all lossless).

We've seen a rough factor of (2.+/-0.5) with lz4 r131 in compression as well with unfiltered data. In my cases we are mostly interested in high compression bandwidth and high compression ratio. lz4 so far gives compression bandwidths up to 1GB/s depending on the quality aspired (of course the compression ratios tend to be lower then).

It was based on those that we settled on Blosc_LZ4HC at level 4, since
we were looking for very fast decompression times, while longer
compression times and slightly larger file size was acceptable up to
certain points. The gzip results are included mostly because that's
what we were using at the time and I wanted them as a comparison, but
we knew we wanted something else. The input for those benchmarks was a
500x300x300 float dataset containing a tomographic 3D image.

to be honest, I am still surprised that hdf5 doesn't contain these state-of-the-art encoders, but rather ships bzip2 et al. which are painfully slow and don't make any account of computer architectures (lz4 is cache aware AFAIK). But hey, coming up with a hdf5 compressor is straight forward after one wrangled with the docs. I just don't know how contributing to hdf5 works.

I might try to dig up the script I used for the benchmark and see if
we still have the input I used, and do a test with lossy ZFP. It could
be very interesting for creating 3D "thumbnails" in our application.

indeed, that would be interesting to see.
Best,
Peter

···

Elvis

P

On 10/28/2016 01:12 PM, Elvis Stansvik wrote:

2016-10-28 1:53 GMT+02:00 Miller, Mark C. <miller86@llnl.gov>:

Hi All,

Just wanted to mention a new HDF5 floating point compression plugin
available on github...

https://github.com/LLNL/H5Z-ZFP

This plugin will come embedded in the next release of the Silo library as
well.

Thanks for the pointer. That's very interesting. I had not heard about
ZFP before. The ability to set a bound on the error in the lossless
case seems very useful.

Do you know if there has been any comparative benchmarks of ZFP
against other compressors?

After some basic benchmarking, we recently settled on Blosc_LZ4HC at
level 4 for our datasets (3D float tomography data), but maybe it
would be worthwhile to look at ZFP as well..

Best regards,
Elvis

--
Mark C. Miller, LLNL

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

faltet · October 28, 2016, 2:33pm

> I second this request big time and would add zstd, if we are already
trying
> out various encoders.

This may not be of interest, and does not include zstd, but I'm
attaching an excerpt from some of the results I got when back when
doing our basic benchmarking of some algorithms (all lossless).

It was based on those that we settled on Blosc_LZ4HC at level 4, since
we were looking for very fast decompression times, while longer
compression times and slightly larger file size was acceptable up to
certain points. The gzip results are included mostly because that's
what we were using at the time and I wanted them as a comparison, but
we knew we wanted something else. The input for those benchmarks was a
500x300x300 float dataset containing a tomographic 3D image.

Zstd was included in Blosc a while ago:

http://blosc.org/blog/zstd-has-just-landed-in-blosc.html

and its performance really shines, even on real data:

http://alimanfoo.github.io/2016/09/21/genotype-compression-benchmark.html

(although here, being only integers of 1 byte, only the BITSHUFFLE filter
is used, but not the faster SHUFFLE).

As Blosc offers the same API for a number of codecs, trying it in
combination with Zstd should be really easy.

I might try to dig up the script I used for the benchmark and see if
we still have the input I used, and do a test with lossy ZFP. It could
be very interesting for creating 3D "thumbnails" in our application.

It would be nice if your benchmark code (and dataset) can be made publicly
available so as to serve to others as a good comparison.

···

2016-10-28 13:59 GMT+02:00 Elvis Stansvik <elvis.stansvik@orexplore.com>:

2016-10-28 13:23 GMT+02:00 Peter Steinbach <steinbach@scionics.de>:

Elvis

>
> P
>
>
> On 10/28/2016 01:12 PM, Elvis Stansvik wrote:
>>
>> 2016-10-28 1:53 GMT+02:00 Miller, Mark C. <miller86@llnl.gov>:
>>>
>>> Hi All,
>>>
>>> Just wanted to mention a new HDF5 floating point compression plugin
>>> available on github...
>>>
>>> https://github.com/LLNL/H5Z-ZFP
>>>
>>> This plugin will come embedded in the next release of the Silo library
as
>>> well.
>>
>>
>> Thanks for the pointer. That's very interesting. I had not heard about
>> ZFP before. The ability to set a bound on the error in the lossless
>> case seems very useful.
>>
>> Do you know if there has been any comparative benchmarks of ZFP
>> against other compressors?
>>
>> After some basic benchmarking, we recently settled on Blosc_LZ4HC at
>> level 4 for our datasets (3D float tomography data), but maybe it
>> would be worthwhile to look at ZFP as well..
>>
>> Best regards,
>> Elvis
>>
>>>
>>> --
>>> Mark C. Miller, LLNL
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> Hdf-forum@lists.hdfgroup.org
>>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.
hdfgroup.org
>>> Twitter: https://twitter.com/hdf5
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@lists.hdfgroup.org
>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>> Twitter: https://twitter.com/hdf5
>>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
Francesc Alted

Peter_Lindstrom · October 31, 2016, 6:03pm

Elvis,

Your compression parameters look suspicious to me:

PARAMS { 5242928 91252346 313532218 -1043792 -937099264 67112167 }

I would start by debugging those. If I understand Mark's filter correctly, the first integer should be 1, 2, 3, or 4.

···

On 10/31/16 09:55, Elvis Stansvik wrote:

Den 31 okt. 2016 5:41 em skrev "Miller, Mark C." <miller86@llnl.gov <mailto:miller86@llnl.gov>>:
>
> Hi Elvis,
>
>>
>> I've successfully tried using the plugin through h5py. I did have a
>> problem with the fixed-accuracy mode however and filed an issue:
>>
>> https://github.com/LLNL/H5Z-ZFP/issues/1
>>
>> It's very likely that I'm doing something wrong though.
>
> Thanks so much!!
>
> I will take a look at the issue you reported later this week.
>
> I may have to rope the researcher who developed ZFP in to help.
>
> AFAICT, I am calling ZFP with correct parameters and, your email confirms you see zfp_stream_set_accuracy() being called.

Alright, no hurry.

I've been in contact with Peter earlier with some other questions. He seems very helpful.

And yes, zfp_stream_set_accuracy seems to be called correctly with the parameters I pass through h5py (same as when I do the equivalent with the zfp command line tool), so I think something else is going on. The dataset type class and dimensions are also correctly identified by the filter plugin.

Elvis

>
> Mark
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
Peter Lindstrom . pl@llnl.gov . http://people.llnl.gov/pl . 925-423-5925

miller86 · October 31, 2016, 6:18pm

Hi Elvis,

The PARAMS dumped from the dataset header are *not* the same cd_values passed in memory between caller and plugin.

The dataset header in the file gets something different; ZFP's header including mode and metadata + H5Z-ZFP plugin version info

The README_MORE file in plugin on github explains this.

Long story short, you can't deduce much from those values dumped by h5dump/h5ls, etc. You'd have to reverse engineer how
ZFP library encodes magic, version, mode and metadata into its header.

Mark

···

--
Mark C. Miller, LLNL

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of Peter Lindstrom <pl@llnl.gov<mailto:pl@llnl.gov>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, October 31, 2016 at 11:03 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] New HDF5 compression plugin

Elvis,

Your compression parameters look suspicious to me:

PARAMS { 5242928 91252346 313532218 -1043792 -937099264 67112167 }

I would start by debugging those. If I understand Mark's filter correctly, the first integer should be 1, 2, 3, or 4.

On 10/31/16 09:55, Elvis Stansvik wrote:

Den 31 okt. 2016 5:41 em skrev "Miller, Mark C." <miller86@llnl.gov<mailto:miller86@llnl.gov>>:

Hi Elvis,

I've successfully tried using the plugin through h5py. I did have a
problem with the fixed-accuracy mode however and filed an issue:

https://github.com/LLNL/H5Z-ZFP/issues/1

It's very likely that I'm doing something wrong though.

Thanks so much!!

I will take a look at the issue you reported later this week.

I may have to rope the researcher who developed ZFP in to help.

AFAICT, I am calling ZFP with correct parameters and, your email confirms you see zfp_stream_set_accuracy() being called.

Alright, no hurry.

I've been in contact with Peter earlier with some other questions. He seems very helpful.

And yes, zfp_stream_set_accuracy seems to be called correctly with the parameters I pass through h5py (same as when I do the equivalent with the zfp command line tool), so I think something else is going on. The dataset type class and dimensions are also correctly identified by the filter plugin.

Elvis

Mark

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
Peter Lindstrom . pl@llnl.gov<mailto:pl@llnl.gov> . http://people.llnl.gov/pl . 925-423-5925

elvis.stansvik · October 28, 2016, 12:38pm

Hi Elvis,

interesting I am mostly looking into 3D optical tomography images (which
exclusively use voxels represented by integers).

This may not be of interest, and does not include zstd, but I'm
attaching an excerpt from some of the results I got when back when
doing our basic benchmarking of some algorithms (all lossless).

We've seen a rough factor of (2.+/-0.5) with lz4 r131 in compression as well
with unfiltered data. In my cases we are mostly interested in high
compression bandwidth and high compression ratio. lz4 so far gives
compression bandwidths up to 1GB/s depending on the quality aspired (of
course the compression ratios tend to be lower then).

It was based on those that we settled on Blosc_LZ4HC at level 4, since
we were looking for very fast decompression times, while longer
compression times and slightly larger file size was acceptable up to
certain points. The gzip results are included mostly because that's
what we were using at the time and I wanted them as a comparison, but
we knew we wanted something else. The input for those benchmarks was a
500x300x300 float dataset containing a tomographic 3D image.

to be honest, I am still surprised that hdf5 doesn't contain these
state-of-the-art encoders, but rather ships bzip2 et al. which are painfully
slow and don't make any account of computer architectures (lz4 is cache
aware AFAIK). But hey, coming up with a hdf5 compressor is straight forward
after one wrangled with the docs. I just don't know how contributing to hdf5
works.

Yea me too, but I believe the HDF5 group has as a goal to open up the
development a bit more, which would be very welcome. So lets hope for
that.

Elvis

···

2016-10-28 14:17 GMT+02:00 Peter Steinbach <steinbach@scionics.de>:

I might try to dig up the script I used for the benchmark and see if
we still have the input I used, and do a test with lossy ZFP. It could
be very interesting for creating 3D "thumbnails" in our application.

indeed, that would be interesting to see.
Best,
Peter

Elvis

P

On 10/28/2016 01:12 PM, Elvis Stansvik wrote:

2016-10-28 1:53 GMT+02:00 Miller, Mark C. <miller86@llnl.gov>:

Hi All,

Just wanted to mention a new HDF5 floating point compression plugin
available on github...

https://github.com/LLNL/H5Z-ZFP

This plugin will come embedded in the next release of the Silo library
as
well.

Thanks for the pointer. That's very interesting. I had not heard about
ZFP before. The ability to set a bound on the error in the lossless
case seems very useful.

Do you know if there has been any comparative benchmarks of ZFP
against other compressors?

After some basic benchmarking, we recently settled on Blosc_LZ4HC at
level 4 for our datasets (3D float tomography data), but maybe it
would be worthwhile to look at ZFP as well..

Best regards,
Elvis

--
Mark C. Miller, LLNL

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

miller86 · October 28, 2016, 3:15pm

to be honest, I am still surprised that hdf5 doesn't contain these
state-of-the-art encoders, but rather ships bzip2 et al. which are
painfully slow and don't make any account of computer architectures (lz4
is cache aware AFAIK). But hey, coming up with a hdf5 compressor is
straight forward after one wrangled with the docs. I just don't know how
contributing to hdf5 works.

FWIW, I think the whole point of the plugin design in HDF5 is to *enable* the community to develop
and support what can potentially be a large variety of complex filters.

I don't think any one team, focused on the core library support could possibly have the resources to
also support a wide variety of compression filters.

The key thing The HDF5 Group *is* doing is managing the filter ids and managing information about
the filters, here...

https://support.hdfgroup.org/services/filters.html#zfp

Mark