Compression filters

Hdf-forum archives
1

Hello all,

here at the DKRZ (German Climate Computing Center) we have to store
large volumes of climate data, some of which is stored in HDF5-files.
So, during the last few month we have been doing some research into
climate data compression.

With very interesting results: We have been able to shrink our test data
set to 38.76% of the original file size. This is a compression factor of
more than 2.5 and it is significantly better than the performance of all
the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc = 66% and
lzma = 46%). Also, we have seen that the lzma-algorithm performs much
better than the other standard algorithms.

Even though we have constructed our methods to fit climate data, the
features we exploited for compression are very general and likely to
apply to other scientific data as well. This is why we are confident
that many of you could profit from these methods as well, and we would
be happy to share our results with the rest of the community.

The filtering mechanism in HDF5 predestines it to be the first place for
us to share our algorithms. But first we would be very interested to see
the lzma algorithm integrated as an optional filtering method, something
that should be very easy to do and offers large benefits to all users.

Since we would be willing to do the necessary work, we just need to know
the (in)formal requirements to integrate a new filter. And, of course,
we would be very interested to hear about other recent work which
adresses compression in HDF5, and to get in touch with whoever works on
it.

Best regards,
Nathanael Hübbe

http://wr.informatik.uni-hamburg.de/
http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe

2

Hello Nathanael,

A while back, I added some application specific compression filters to a
library I support that uses HDF5. I spent a lot of time reading the
following...

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html#Compression-Register

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFilter

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-FilterBehavior

Also, if you would like to see examples of how I did it, you are welcome
to download

https://wci.llnl.gov/codes/silo/silo-4.8/silo-4.8-bsd-smalltest.tar.gz

Untar it and cd to src/hdf5_drv and edit silo_hdf5.c. If you look around
in there for any 'H5Z' stuff, you will find logic that implements two
different types of user-defined compression filters in Silo.

Hope that helps.

Mark

···

On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:

Hello all,

here at the DKRZ (German Climate Computing Center) we have to store
large volumes of climate data, some of which is stored in HDF5-files.
So, during the last few month we have been doing some research into
climate data compression.

With very interesting results: We have been able to shrink our test data
set to 38.76% of the original file size. This is a compression factor of
more than 2.5 and it is significantly better than the performance of all
the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc = 66% and
lzma = 46%). Also, we have seen that the lzma-algorithm performs much
better than the other standard algorithms.

Even though we have constructed our methods to fit climate data, the
features we exploited for compression are very general and likely to
apply to other scientific data as well. This is why we are confident
that many of you could profit from these methods as well, and we would
be happy to share our results with the rest of the community.

The filtering mechanism in HDF5 predestines it to be the first place for
us to share our algorithms. But first we would be very interested to see
the lzma algorithm integrated as an optional filtering method, something
that should be very easy to do and offers large benefits to all users.

Since we would be willing to do the necessary work, we just need to know
the (in)formal requirements to integrate a new filter. And, of course,
we would be very interested to hear about other recent work which
adresses compression in HDF5, and to get in touch with whoever works on
it.

Best regards,
Nathanael H�bbe

http://wr.informatik.uni-hamburg.de/
http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

3

Hi Nathanael,

Hello all,

here at the DKRZ (German Climate Computing Center) we have to store
large volumes of climate data, some of which is stored in HDF5-files.
So, during the last few month we have been doing some research into
climate data compression.

With very interesting results: We have been able to shrink our test data
set to 38.76% of the original file size. This is a compression factor of
more than 2.5 and it is significantly better than the performance of all
the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc = 66% and
lzma = 46%). Also, we have seen that the lzma-algorithm performs much
better than the other standard algorithms.

Even though we have constructed our methods to fit climate data, the
features we exploited for compression are very general and likely to
apply to other scientific data as well. This is why we are confident
that many of you could profit from these methods as well, and we would
be happy to share our results with the rest of the community.

The filtering mechanism in HDF5 predestines it to be the first place for
us to share our algorithms. But first we would be very interested to see
the lzma algorithm integrated as an optional filtering method, something
that should be very easy to do and offers large benefits to all users.

Since we would be willing to do the necessary work, we just need to know
the (in)formal requirements to integrate a new filter. And, of course,
we would be very interested to hear about other recent work which
adresses compression in HDF5, and to get in touch with whoever works on
it.

Any new future adds to software maintenance and we (The HDF Group) have to be very careful about it.

We have been working on a process of accepting patches and new features from the community, but there are a few issues (like criterion of acceptence, IP) that have to be resolved. I.e., at this point we do not have criterion and formal requirements for integrating new features.

But we do have a procedure for registering new filters with us. It is simple:

To request a filter identifier please contact help@hdfgroup.org with the following information

# Contact information for developer requesting the new identifier
# Short description of the new filter
# Links to any relevant information including licensing information

Information about the 3rd party filters that are registered with us will be on the Web (TBD; we are working on it; it was on our PBWiki and we don't use it anymore).

Currently it is:

305 LZO (used by PyTables)
307 BZIP2 (used by PyTables)
32000 LZF (used by H5Py)
32001 BLOSC (used by PyTables)

Please let me know if you have any questions.

Elena

···

On Oct 11, 2011, at 10:18 AM, Nathanael Huebbe wrote:

Best regards,
Nathanael Hübbe

http://wr.informatik.uni-hamburg.de/
http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

4

Sorry my last email was a bit off topic. I guess I latched on to the
"...we just need to know (in)formal requirements to integrate a new
filter" part.

So, related to this, I thought I recall a move afoot a couple of years
back now to formalize submission, repository of 3rd party HDF5 'filters'
of various sorts and compression in particular. What happened with that?
I don't recall seeing much response on the forum to it but this sounds
like a good test case :wink:

Mark

···

On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:

Hello all,

here at the DKRZ (German Climate Computing Center) we have to store
large volumes of climate data, some of which is stored in HDF5-files.
So, during the last few month we have been doing some research into
climate data compression.

With very interesting results: We have been able to shrink our test data
set to 38.76% of the original file size. This is a compression factor of
more than 2.5 and it is significantly better than the performance of all
the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc = 66% and
lzma = 46%). Also, we have seen that the lzma-algorithm performs much
better than the other standard algorithms.

Even though we have constructed our methods to fit climate data, the
features we exploited for compression are very general and likely to
apply to other scientific data as well. This is why we are confident
that many of you could profit from these methods as well, and we would
be happy to share our results with the rest of the community.

The filtering mechanism in HDF5 predestines it to be the first place for
us to share our algorithms. But first we would be very interested to see
the lzma algorithm integrated as an optional filtering method, something
that should be very easy to do and offers large benefits to all users.

Since we would be willing to do the necessary work, we just need to know
the (in)formal requirements to integrate a new filter. And, of course,
we would be very interested to hear about other recent work which
adresses compression in HDF5, and to get in touch with whoever works on
it.

Best regards,
Nathanael H�bbe

http://wr.informatik.uni-hamburg.de/
http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

5

Nathaniel,

Even though we have constructed our methods to fit climate data, the
features we exploited for compression are very general and likely to
apply to other scientific data as well. This is why we are confident
that many of you could profit from these methods as well, and we would
be happy to share our results with the rest of the community.
<<

Can you supply a little more information about what the data is and what the compression is doing that's special?

Is it ECHAM data for example? Are you compressing a single field array, or a whole collection of arrays, individually or combined into a larger multidimensional dataset. Are they high variance datasets of slowly varying fields?

Did you apply any other filters such as the shuffle filter before compressing, and is the compression lossless (assume yes).
If you switched to a lossy compression, by first packing the data into a range (scale-offset filter?), how would that affect the final compression? Have you looked into this?

Are there more details of this kind available? if so, may I see them please. We have users with excessive data storage needs and any progress towards reducing it is welcome.

thanks

JB.
PS. others suggested sites to host code and I can recommend our hpc forge site where our VFD is hosted
https://hpcforge.org/projects/h5fddsm/
if this is of interest, I'd be happy to setup things for you.

6

Hi Nathanael,

One thing to keep in mind is that HDF5 is BSD licensed so you might
want to make your compression code BSD licensed as well, if that is
possible at your organization.

Cheers,

Dana

7

Out of curiosity, did you try the szip filter provided with HDF5? (If
you are able to use it under the licensing terms.)

We took as close a look as possible at the algorithm but concluded that
it cannot hope to compete with lzma due to the suboptimality of the rice
coder used for entropy coding. Range coders, like the one lzma uses, are
already optimal in terms of storage space while rice coding makes
assumptions about the frequency of symbols that are generally not met.

Hello Nathanael,

A while back, I added some application specific compression filters to

a

library I support that uses HDF5. I spent a lot of time reading the
following...

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5Z.html#Compression-Register

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFilter

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-FilterBehavior

Yes, I know these pages by heart already (well, almost), since we
already have a moded version of the h5repack utility and an extremely
minimalistic API (one function call) to include support into any
application using HDF5 files. You can find a patch with the modified
h5repack utility at:
http://files.wr.informatik.uni-hamburg.de/hdf5-dkrz-compressor.patch

So the question is not, how we can use our algorithm in HDF5 files, we
can already do it; the question is, whether you want support for lzma
and/or our algorithm in the standard library, which we would be glad to
contribute.

Best regards,
Nathanael Hübbe

···

On Tue, 2011-10-11 at 12:33 -0400, Zaak Beekman wrote:
On Wed, 2011-10-12 at 00:07 -0700, Mark Miller wrote:

8

I think it would be nice if a 3rd party filter is loaded dynamically
using dlopen.
The name of the filter needs to be reflected in the library name in
some standard way. The init function of the library can register the
filter in HDF5's filter registry. I think it only requires a small
addition to the way HDF5 finds a filter.
In this way one the filter repository can be a set of shared libraries
or dlls; one does not need to change HDF5 to use a new filter. Also all
HDF5 tools can work with all filters.

This is the way python works and how query languages execute user
defined functions.
I've used it myself successfully several times.

Cheers,
Ger

Mark Miller <miller86@llnl.gov> 10/13/2011 4:32 AM >>>

Sorry my last email was a bit off topic. I guess I latched on to the
"...we just need to know (in)formal requirements to integrate a new
filter" part.

So, related to this, I thought I recall a move afoot a couple of years
back now to formalize submission, repository of 3rd party HDF5
'filters'
of various sorts and compression in particular. What happened with
that?
I don't recall seeing much response on the forum to it but this sounds
like a good test case :wink:

Mark

Hello all,

here at the DKRZ (German Climate Computing Center) we have to store
large volumes of climate data, some of which is stored in

HDF5-files.

So, during the last few month we have been doing some research into
climate data compression.

With very interesting results: We have been able to shrink our test

data

set to 38.76% of the original file size. This is a compression factor

of

more than 2.5 and it is significantly better than the performance of

all

the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc = 66%

and

lzma = 46%). Also, we have seen that the lzma-algorithm performs

much

better than the other standard algorithms.

Even though we have constructed our methods to fit climate data, the
features we exploited for compression are very general and likely to
apply to other scientific data as well. This is why we are confident
that many of you could profit from these methods as well, and we

would

be happy to share our results with the rest of the community.

The filtering mechanism in HDF5 predestines it to be the first place

for

us to share our algorithms. But first we would be very interested to

see

the lzma algorithm integrated as an optional filtering method,

something

that should be very easy to do and offers large benefits to all

users.

Since we would be willing to do the necessary work, we just need to

know

the (in)formal requirements to integrate a new filter. And, of

course,

we would be very interested to hear about other recent work which
adresses compression in HDF5, and to get in touch with whoever works

on

···

On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:

it.

Best regards,
Nathanael Hübbe

http://wr.informatik.uni-hamburg.de/
http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

9

There is a public source code repository for HDF5-addons, hosted
at Origo, which is something like the Swiss version of Sourceforge:

http://hdf5-addons.origo.ethz.ch/

This one could be used for such filters as well. As of now it just
contains the HDF5 Streaming VFD, which was outsourced from HDF5 1.6.4
some time ago. Origo is a pretty nice platform offering a couple
of useful tools beyond just SVN management. Currently it's Quincey
and myself being project managers there, we could easily add other
project members to allow them adding their 3rd party extensions
there. This platform could also be a first step of publishing
HDF5 addons before official integration into HDF5 itself, in case
such is on the roadmap, which may or may not be the case for
HDF5 addons. If licensing schemes are incompatible, it might be
better to have another, accessible place - Origo is very flexible,
in contrast to sourceforge it does not require open source licensing
for projects hosted there, so it can also host commercial code
which is still publicly available under a non-OS license.

  Werner

···

On Thu, 13 Oct 2011 04:32:33 +0200, Mark Miller <miller86@llnl.gov> wrote:

Sorry my last email was a bit off topic. I guess I latched on to the
"...we just need to know (in)formal requirements to integrate a new
filter" part.

So, related to this, I thought I recall a move afoot a couple of years
back now to formalize submission, repository of 3rd party HDF5 'filters'
of various sorts and compression in particular. What happened with that?
I don't recall seeing much response on the forum to it but this sounds
like a good test case :wink:

Mark

On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:

Hello all,

here at the DKRZ (German Climate Computing Center) we have to store
large volumes of climate data, some of which is stored in HDF5-files.
So, during the last few month we have been doing some research into
climate data compression.

With very interesting results: We have been able to shrink our test data
set to 38.76% of the original file size. This is a compression factor of
more than 2.5 and it is significantly better than the performance of all
the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc = 66% and
lzma = 46%). Also, we have seen that the lzma-algorithm performs much
better than the other standard algorithms.

Even though we have constructed our methods to fit climate data, the
features we exploited for compression are very general and likely to
apply to other scientific data as well. This is why we are confident
that many of you could profit from these methods as well, and we would
be happy to share our results with the rest of the community.

The filtering mechanism in HDF5 predestines it to be the first place for
us to share our algorithms. But first we would be very interested to see
the lzma algorithm integrated as an optional filtering method, something
that should be very easy to do and offers large benefits to all users.

Since we would be willing to do the necessary work, we just need to know
the (in)formal requirements to integrate a new filter. And, of course,
we would be very interested to hear about other recent work which
adresses compression in HDF5, and to get in touch with whoever works on
it.

Best regards,
Nathanael Hübbe

http://wr.informatik.uni-hamburg.de/
http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

10

But we do have a procedure for registering new filters with us. It is
simple:

To request a filter identifier please contact help@hdfgroup.org
with the following information

# Contact information for developer requesting the new identifier
# Short description of the new filter
# Links to any relevant information including licensing
information

Thanks, I will do this. As a matter of fact, the software is already
under a two clause BSD style license, so no compatibility concerns here.

···

On Wed, 2011-10-12 at 09:58 -0500, Elena Pourmal wrote:

On Thu, 2011-10-13 at 07:51 +0200, Ger van Diepen wrote:

I think it would be nice if a 3rd party filter is loaded dynamically
using dlopen.

The name of the filter needs to be reflected in the library name in
some standard way. The init function of the library can register the
filter in HDF5's filter registry. I think it only requires a small
addition to the way HDF5 finds a filter.

In this way one the filter repository can be a set of shared libraries
or dlls; one does not need to change HDF5 to use a new filter. Also
all HDF5 tools can work with all filters.

This sounds great. If such a scheme would be implemented, we would
certainly provide a pure lzma filter as well as our own algorithm.

Best regards,
Nathanael

11

Well, my recollection is something that I think Elena Pourmal was trying
to start where source code for (open source) 3rd party filters somehow
got submitted to HDF Group's web pages along with example data files
containing the 'filtered' data and information about the filter such as
Point of contact, purpose, performance characteristics, etc.

I like the idea of making it 'easy' to install such 3rd party filters
with (or next to) the main HDF5 library so they can be loaded as needed.
But, that is a step beyond what I was trying to open a dialog about
here :wink:

Mark

···

On Wed, 2011-10-12 at 22:51 -0700, Ger van Diepen wrote:

I think it would be nice if a 3rd party filter is loaded dynamically
using dlopen.

The name of the filter needs to be reflected in the library name in
some standard way. The init function of the library can register the
filter in HDF5's filter registry. I think it only requires a small
addition to the way HDF5 finds a filter.

In this way one the filter repository can be a set of shared libraries
or dlls; one does not need to change HDF5 to use a new filter. Also
all HDF5 tools can work with all filters.

This is the way python works and how query languages execute user
defined functions.

I've used it myself successfully several times.

Cheers,

Ger

>>> Mark Miller <miller86@llnl.gov> 10/13/2011 4:32 AM >>>
Sorry my last email was a bit off topic. I guess I latched on to the
"...we just need to know (in)formal requirements to integrate a new
filter" part.

So, related to this, I thought I recall a move afoot a couple of years
back now to formalize submission, repository of 3rd party HDF5
'filters'
of various sorts and compression in particular. What happened with
that?
I don't recall seeing much response on the forum to it but this sounds
like a good test case :wink:

Mark

On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:
> Hello all,
>
> here at the DKRZ (German Climate Computing Center) we have to store
> large volumes of climate data, some of which is stored in
HDF5-files.
> So, during the last few month we have been doing some research into
> climate data compression.
>
> With very interesting results: We have been able to shrink our test
data
> set to 38.76% of the original file size. This is a compression
factor of
> more than 2.5 and it is significantly better than the performance of
all
> the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc = 66%
and
> lzma = 46%). Also, we have seen that the lzma-algorithm performs
much
> better than the other standard algorithms.
>
> Even though we have constructed our methods to fit climate data, the
> features we exploited for compression are very general and likely to
> apply to other scientific data as well. This is why we are confident
> that many of you could profit from these methods as well, and we
would
> be happy to share our results with the rest of the community.
>
> The filtering mechanism in HDF5 predestines it to be the first place
for
> us to share our algorithms. But first we would be very interested to
see
> the lzma algorithm integrated as an optional filtering method,
something
> that should be very easy to do and offers large benefits to all
users.
>
> Since we would be willing to do the necessary work, we just need to
know
> the (in)formal requirements to integrate a new filter. And, of
course,
> we would be very interested to hear about other recent work which
> adresses compression in HDF5, and to get in touch with whoever works
on
> it.
>
> Best regards,
> Nathanael H�bbe
>
> http://wr.informatik.uni-hamburg.de/
> http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

12

Adding support for runtime plugins may impose its own portability issues,
needing to handle incompatibilities of libraries (was the plugin compiled
with the same library as the library trying to load it, etc.), and some
additional management schemes (such as search path of directories for
plugins). All solvable, but certainly requires some time. But with HDF5
there might also be the option to embed a filter into HDF5 itself,
as a binary, platform-dependent code which is stored in an "unfiltered"
HDF5 dataset in a predefined location. This could allow an HDF5 file
to bring its own filter with it, precompiled for a set of supported
platforms. Maybe this would make it easier? Just an idea. Of course,
would require some kind of certification scheme to avoid opening the
path for viruses and trojans, but the same is the case with runtime
plugins as well.

  Werner

···

On Thu, 13 Oct 2011 08:11:47 +0200, Mark Miller <miller86@llnl.gov> wrote:

Well, my recollection is something that I think Elena Pourmal was trying
to start where source code for (open source) 3rd party filters somehow
got submitted to HDF Group's web pages along with example data files
containing the 'filtered' data and information about the filter such as
Point of contact, purpose, performance characteristics, etc.

I like the idea of making it 'easy' to install such 3rd party filters
with (or next to) the main HDF5 library so they can be loaded as needed.
But, that is a step beyond what I was trying to open a dialog about
here :wink:

Mark

On Wed, 2011-10-12 at 22:51 -0700, Ger van Diepen wrote:

I think it would be nice if a 3rd party filter is loaded dynamically
using dlopen.

The name of the filter needs to be reflected in the library name in
some standard way. The init function of the library can register the
filter in HDF5's filter registry. I think it only requires a small
addition to the way HDF5 finds a filter.

In this way one the filter repository can be a set of shared libraries
or dlls; one does not need to change HDF5 to use a new filter. Also
all HDF5 tools can work with all filters.

This is the way python works and how query languages execute user
defined functions.

I've used it myself successfully several times.

Cheers,

Ger

>>> Mark Miller <miller86@llnl.gov> 10/13/2011 4:32 AM >>>
Sorry my last email was a bit off topic. I guess I latched on to the
"...we just need to know (in)formal requirements to integrate a new
filter" part.

So, related to this, I thought I recall a move afoot a couple of years
back now to formalize submission, repository of 3rd party HDF5
'filters'
of various sorts and compression in particular. What happened with
that?
I don't recall seeing much response on the forum to it but this sounds
like a good test case :wink:

Mark

On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:
> Hello all,
>
> here at the DKRZ (German Climate Computing Center) we have to store
> large volumes of climate data, some of which is stored in
HDF5-files.
> So, during the last few month we have been doing some research into
> climate data compression.
>
> With very interesting results: We have been able to shrink our test
data
> set to 38.76% of the original file size. This is a compression
factor of
> more than 2.5 and it is significantly better than the performance of
all
> the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc = 66%
and
> lzma = 46%). Also, we have seen that the lzma-algorithm performs
much
> better than the other standard algorithms.
>
> Even though we have constructed our methods to fit climate data, the
> features we exploited for compression are very general and likely to
> apply to other scientific data as well. This is why we are confident
> that many of you could profit from these methods as well, and we
would
> be happy to share our results with the rest of the community.
>
> The filtering mechanism in HDF5 predestines it to be the first place
for
> us to share our algorithms. But first we would be very interested to
see
> the lzma algorithm integrated as an optional filtering method,
something
> that should be very easy to do and offers large benefits to all
users.
>
> Since we would be willing to do the necessary work, we just need to
know
> the (in)formal requirements to integrate a new filter. And, of
course,
> we would be very interested to hear about other recent work which
> adresses compression in HDF5, and to get in touch with whoever works
on
> it.
>
> Best regards,
> Nathanael Hübbe
>
> http://wr.informatik.uni-hamburg.de/
> http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

13

Mark and All,

Well, my recollection is something that I think Elena Pourmal was trying
to start where source code for (open source) 3rd party filters somehow
got submitted to HDF Group's web pages along with example data files
containing the 'filtered' data and information about the filter such as
Point of contact, purpose, performance characteristics, etc.

This information is available at The HDF Group Web site http://www.hdfgroup.org/services/contributions.html

In the past, developers who supported the filters had access to PBWiki and could update corresponding information. Currently this functionality is missing.
Please contact help@hdfgroup.org if you need help with updates.

I like the idea of making it 'easy' to install such 3rd party filters
with (or next to) the main HDF5 library so they can be loaded as needed.
But, that is a step beyond what I was trying to open a dialog about
here :wink:

This will be a great feature and we will be happy to consider and accept the patches to the library.

Elena

···

On Oct 13, 2011, at 1:11 AM, Mark Miller wrote:

Mark

On Wed, 2011-10-12 at 22:51 -0700, Ger van Diepen wrote:

I think it would be nice if a 3rd party filter is loaded dynamically
using dlopen.

The name of the filter needs to be reflected in the library name in
some standard way. The init function of the library can register the
filter in HDF5's filter registry. I think it only requires a small
addition to the way HDF5 finds a filter.

In this way one the filter repository can be a set of shared libraries
or dlls; one does not need to change HDF5 to use a new filter. Also
all HDF5 tools can work with all filters.

This is the way python works and how query languages execute user
defined functions.

I've used it myself successfully several times.

Cheers,

Ger

Mark Miller <miller86@llnl.gov> 10/13/2011 4:32 AM >>>

Sorry my last email was a bit off topic. I guess I latched on to the
"...we just need to know (in)formal requirements to integrate a new
filter" part.

So, related to this, I thought I recall a move afoot a couple of years
back now to formalize submission, repository of 3rd party HDF5
'filters'
of various sorts and compression in particular. What happened with
that?
I don't recall seeing much response on the forum to it but this sounds
like a good test case :wink:

Mark

On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:

Hello all,

here at the DKRZ (German Climate Computing Center) we have to store
large volumes of climate data, some of which is stored in

HDF5-files.

So, during the last few month we have been doing some research into
climate data compression.

With very interesting results: We have been able to shrink our test

data

set to 38.76% of the original file size. This is a compression

factor of

more than 2.5 and it is significantly better than the performance of

all

the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc = 66%

and

lzma = 46%). Also, we have seen that the lzma-algorithm performs

much

better than the other standard algorithms.

Even though we have constructed our methods to fit climate data, the
features we exploited for compression are very general and likely to
apply to other scientific data as well. This is why we are confident
that many of you could profit from these methods as well, and we

would

be happy to share our results with the rest of the community.

The filtering mechanism in HDF5 predestines it to be the first place

for

us to share our algorithms. But first we would be very interested to

see

the lzma algorithm integrated as an optional filtering method,

something

that should be very easy to do and offers large benefits to all

users.

Since we would be willing to do the necessary work, we just need to

know

the (in)formal requirements to integrate a new filter. And, of

course,

we would be very interested to hear about other recent work which
adresses compression in HDF5, and to get in touch with whoever works

on

it.

Best regards,
Nathanael Hübbe

http://wr.informatik.uni-hamburg.de/
http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

14

Hi all,

This information is available at The HDF Group Web site http://www.hdfgroup.org/services/contributions.html

In the past, developers who supported the filters had access to PBWiki and could update corresponding information. Currently this functionality is missing.
Please contact help@hdfgroup.org if you need help with updates.

I like the idea of making it 'easy' to install such 3rd party filters
with (or next to) the main HDF5 library so they can be loaded as needed.
But, that is a step beyond what I was trying to open a dialog about
here :wink:

While the wiki is down you can find some information on the h5py LZF
filter here: http://h5py.alfven.org/lzf

As people no doubt know, you can write a filter on your own and
register it with HDF5 at runtime using H5Zregister. No modifications
have to be made to HDF5. However, you have to specify a filter ID,
which ends up stored in the file. So to prevent chaos, the HDF Group
serves as an authority for issuing filter IDs.

The h5py LZF filter is actually implemented as a standalone C program
which is statically linked into h5py. It can be used in other C/C++
programs or as a shared library. It's included in the h5py
distribution tarball under the directory "lzf". There's also an
example C program which uses the filter. Everything's under the BSD
license so feel free to take what you wish.

You can see the filter in Google Code browser view here:

http://code.google.com/p/h5py/source/browse/#hg%2Flzf

The most recent h5py tarball (with the filter) is here:

http://h5py.googlecode.com/files/h5py-2.0.1.tar.gz

Happy filtering,
Andrew

15

may I send you a patch to you directly ?

···

On 13/10/11 18:13, Elena Pourmal wrote:

This will be a great feature and we will be happy to consider and accept the patches to the library.

Elena

16

Out of curiosity, did you try the szip filter provided with HDF5? (If you
are able to use it under the licensing terms.)

Izaak Beekman

···

===================================
(301)244-9367
Princeton University Doctoral Candidate
Mechanical and Aerospace Engineering
ibeekman@princeton.edu

UMD-CP Visiting Graduate Student
Aerospace Engineering
ibeekman@umiacs.umd.edu
ibeekman@umd.edu

17

Hello List:

I think it would be nice if a 3rd party filter is loaded dynamically using dlopen.

It can be done easily actually:
I use some filters of mime in this way as xz comprressor.

hth,
Jerome

···

-------- Original Message --------
Subject: Re: [Hdf-forum] Compression filters
Date: Thu, 13 Oct 2011 10:57:29 +0200
From: Jerome BENOIT <jgmb65@rezozer.net>
Reply-To: jgmb65@rezozer.net
Organization: none
To: hdf-forum@hdfgroup.org

On 13/10/11 07:51, Ger van Diepen wrote:

The name of the filter needs to be reflected in the library name in some standard way. The init function of the library can register the filter in HDF5's filter registry. I think it only requires a small addition to the way HDF5 finds a filter.

In this way one the filter repository can be a set of shared libraries or dlls; one does not need to change HDF5 to use a new filter. Also all HDF5 tools can work with all filters.

This is the way python works and how query languages execute user defined functions.

I've used it myself successfully several times.

Cheers,

Ger

>>> Mark Miller <miller86@llnl.gov> 10/13/2011 4:32 AM >>>
Sorry my last email was a bit off topic. I guess I latched on to the
"...we just need to know (in)formal requirements to integrate a new
filter" part.

So, related to this, I thought I recall a move afoot a couple of years
back now to formalize submission, repository of 3rd party HDF5 'filters'
of various sorts and compression in particular. What happened with that?
I don't recall seeing much response on the forum to it but this sounds
like a good test case :wink:

Mark

On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:
> Hello all,
>
> here at the DKRZ (German Climate Computing Center) we have to store
> large volumes of climate data, some of which is stored in HDF5-files.
> So, during the last few month we have been doing some research into
> climate data compression.
>
> With very interesting results: We have been able to shrink our test data
> set to 38.76% of the original file size. This is a compression factor of
> more than 2.5 and it is significantly better than the performance ofall
> the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc =66% and
> lzma = 46%). Also, we have seen that the lzma-algorithm performs much
> better than the other standard algorithms.
>
> Even though we have constructed our methods to fit climate data, the
> features we exploited for compression are very general and likely to
> apply to other scientific data as well. This is why we are confident
> that many of you could profit from these methods as well, and we would
> be happy to share our results with the rest of the community.
>
> The filtering mechanism in HDF5 predestines it to be the first placefor
> us to share our algorithms. But first we would be very interested tosee
> the lzma algorithm integrated as an optional filtering method, something
> that should be very easy to do and offers large benefits to all users.
>
> Since we would be willing to do the necessary work, we just need to know
> the (in)formal requirements to integrate a new filter. And, of course,
> we would be very interested to hear about other recent work which
> adresses compression in HDF5, and to get in touch with whoever workson
> it.
>
> Best regards,
> Nathanael Hübbe
>
> http://wr.informatik.uni-hamburg.de/
> http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

18

Is there a place where a patch can be sent ?

···

On 13/10/11 10:58, Jerome BENOIT wrote:

-------- Original Message --------
Subject: Re: [Hdf-forum] Compression filters
Date: Thu, 13 Oct 2011 10:57:29 +0200
From: Jerome BENOIT <jgmb65@rezozer.net>
Reply-To: jgmb65@rezozer.net
Organization: none
To: hdf-forum@hdfgroup.org

Hello List:

On 13/10/11 07:51, Ger van Diepen wrote:

I think it would be nice if a 3rd party filter is loaded dynamically using dlopen.

It can be done easily actually:
I use some filters of mime in this way as xz comprressor.

hth,
Jerome

The name of the filter needs to be reflected in the library name in some standard way. The init function of the library can register the filter in HDF5's filter registry. I think it only requires a small addition to the way HDF5 finds a filter.

In this way one the filter repository can be a set of shared libraries or dlls; one does not need to change HDF5 to use a new filter. Also all HDF5 tools can work with all filters.

This is the way python works and how query languages execute user defined functions.

I've used it myself successfully several times.

Cheers,

Ger

>>> Mark Miller <miller86@llnl.gov> 10/13/2011 4:32 AM >>>
Sorry my last email was a bit off topic. I guess I latched on to the
"...we just need to know (in)formal requirements to integrate a new
filter" part.

So, related to this, I thought I recall a move afoot a couple of years
back now to formalize submission, repository of 3rd party HDF5 'filters'
of various sorts and compression in particular. What happened with that?
I don't recall seeing much response on the forum to it but this sounds
like a good test case :wink:

Mark

On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:
> Hello all,
>
> here at the DKRZ (German Climate Computing Center) we have to store
> large volumes of climate data, some of which is stored in HDF5-files.
> So, during the last few month we have been doing some research into
> climate data compression.
>
> With very interesting results: We have been able to shrink our test data
> set to 38.76% of the original file size. This is a compression factor of
> more than 2.5 and it is significantly better than the performance ofall
> the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc =66% and
> lzma = 46%). Also, we have seen that the lzma-algorithm performs much
> better than the other standard algorithms.
>
> Even though we have constructed our methods to fit climate data, the
> features we exploited for compression are very general and likely to
> apply to other scientific data as well. This is why we are confident
> that many of you could profit from these methods as well, and we would
> be happy to share our results with the rest of the community.
>
> The filtering mechanism in HDF5 predestines it to be the first placefor
> us to share our algorithms. But first we would be very interested tosee
> the lzma algorithm integrated as an optional filtering method, something
> that should be very easy to do and offers large benefits to all users.
>
> Since we would be willing to do the necessary work, we just need to know
> the (in)formal requirements to integrate a new filter. And, of course,
> we would be very interested to hear about other recent work which
> adresses compression in HDF5, and to get in touch with whoever workson
> it.
>
> Best regards,
> Nathanael Hübbe
>
> http://wr.informatik.uni-hamburg.de/
> http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

19

Please sned it to help@hdfgroup.org

Elena

···

On Oct 14, 2011, at 6:37 AM, Jerome BENOIT wrote:

Is there a place where a patch can be sent ?

On 13/10/11 10:58, Jerome BENOIT wrote:

-------- Original Message --------
Subject: Re: [Hdf-forum] Compression filters
Date: Thu, 13 Oct 2011 10:57:29 +0200
From: Jerome BENOIT <jgmb65@rezozer.net>
Reply-To: jgmb65@rezozer.net
Organization: none
To: hdf-forum@hdfgroup.org

Hello List:

On 13/10/11 07:51, Ger van Diepen wrote:

I think it would be nice if a 3rd party filter is loaded dynamically using dlopen.

It can be done easily actually:
I use some filters of mime in this way as xz comprressor.

hth,
Jerome

The name of the filter needs to be reflected in the library name in some standard way. The init function of the library can register the filter in HDF5's filter registry. I think it only requires a small addition to the way HDF5 finds a filter.

In this way one the filter repository can be a set of shared libraries or dlls; one does not need to change HDF5 to use a new filter. Also all HDF5 tools can work with all filters.

This is the way python works and how query languages execute user defined functions.

I've used it myself successfully several times.

Cheers,

Ger

>>> Mark Miller <miller86@llnl.gov> 10/13/2011 4:32 AM >>>
Sorry my last email was a bit off topic. I guess I latched on to the
"...we just need to know (in)formal requirements to integrate a new
filter" part.

So, related to this, I thought I recall a move afoot a couple of years
back now to formalize submission, repository of 3rd party HDF5 'filters'
of various sorts and compression in particular. What happened with that?
I don't recall seeing much response on the forum to it but this sounds
like a good test case :wink:

Mark

On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:
> Hello all,
>
> here at the DKRZ (German Climate Computing Center) we have to store
> large volumes of climate data, some of which is stored in HDF5-files.
> So, during the last few month we have been doing some research into
> climate data compression.
>
> With very interesting results: We have been able to shrink our test data
> set to 38.76% of the original file size. This is a compression factor of
> more than 2.5 and it is significantly better than the performance ofall
> the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc =66% and
> lzma = 46%). Also, we have seen that the lzma-algorithm performs much
> better than the other standard algorithms.
>
> Even though we have constructed our methods to fit climate data, the
> features we exploited for compression are very general and likely to
> apply to other scientific data as well. This is why we are confident
> that many of you could profit from these methods as well, and we would
> be happy to share our results with the rest of the community.
>
> The filtering mechanism in HDF5 predestines it to be the first placefor
> us to share our algorithms. But first we would be very interested tosee
> the lzma algorithm integrated as an optional filtering method, something
> that should be very easy to do and offers large benefits to all users.
>
> Since we would be willing to do the necessary work, we just need to know
> the (in)formal requirements to integrate a new filter. And, of course,
> we would be very interested to hear about other recent work which
> adresses compression in HDF5, and to get in touch with whoever workson
> it.
>
> Best regards,
> Nathanael Hübbe
>
> http://wr.informatik.uni-hamburg.de/
> http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

20

I have just sent it.

···

On 14/10/11 16:11, Elena Pourmal wrote:

Please sned it to help@hdfgroup.org

Elena
On Oct 14, 2011, at 6:37 AM, Jerome BENOIT wrote:

Is there a place where a patch can be sent ?

On 13/10/11 10:58, Jerome BENOIT wrote:

-------- Original Message --------
Subject: Re: [Hdf-forum] Compression filters
Date: Thu, 13 Oct 2011 10:57:29 +0200
From: Jerome BENOIT<jgmb65@rezozer.net>
Reply-To: jgmb65@rezozer.net
Organization: none
To: hdf-forum@hdfgroup.org

Hello List:

On 13/10/11 07:51, Ger van Diepen wrote:

I think it would be nice if a 3rd party filter is loaded dynamically using dlopen.

It can be done easily actually:
I use some filters of mime in this way as xz comprressor.

hth,
Jerome

The name of the filter needs to be reflected in the library name in some standard way. The init function of the library can register the filter in HDF5's filter registry. I think it only requires a small addition to the way HDF5 finds a filter.

In this way one the filter repository can be a set of shared libraries or dlls; one does not need to change HDF5 to use a new filter. Also all HDF5 tools can work with all filters.

This is the way python works and how query languages execute user defined functions.

I've used it myself successfully several times.

Cheers,

Ger

Mark Miller<miller86@llnl.gov> 10/13/2011 4:32 AM>>>

Sorry my last email was a bit off topic. I guess I latched on to the
"...we just need to know (in)formal requirements to integrate a new
filter" part.

So, related to this, I thought I recall a move afoot a couple of years
back now to formalize submission, repository of 3rd party HDF5 'filters'
of various sorts and compression in particular. What happened with that?
I don't recall seeing much response on the forum to it but this sounds
like a good test case :wink:

Mark

On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:

Hello all,

here at the DKRZ (German Climate Computing Center) we have to store
large volumes of climate data, some of which is stored in HDF5-files.
So, during the last few month we have been doing some research into
climate data compression.

With very interesting results: We have been able to shrink our test data
set to 38.76% of the original file size. This is a compression factor of
more than 2.5 and it is significantly better than the performance ofall
the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc =66% and
lzma = 46%). Also, we have seen that the lzma-algorithm performs much
better than the other standard algorithms.

Even though we have constructed our methods to fit climate data, the
features we exploited for compression are very general and likely to
apply to other scientific data as well. This is why we are confident
that many of you could profit from these methods as well, and we would
be happy to share our results with the rest of the community.

The filtering mechanism in HDF5 predestines it to be the first placefor
us to share our algorithms. But first we would be very interested tosee
the lzma algorithm integrated as an optional filtering method, something
that should be very easy to do and offers large benefits to all users.

Since we would be willing to do the necessary work, we just need to know
the (in)formal requirements to integrate a new filter. And, of course,
we would be very interested to hear about other recent work which
adresses compression in HDF5, and to get in touch with whoever workson
it.

Best regards,
Nathanael Hübbe

http://wr.informatik.uni-hamburg.de/
http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org