I think it would be nice if a 3rd party filter is loaded dynamically
using dlopen.
The name of the filter needs to be reflected in the library name in
some standard way. The init function of the library can register the
filter in HDF5's filter registry. I think it only requires a small
addition to the way HDF5 finds a filter.
In this way one the filter repository can be a set of shared libraries
or dlls; one does not need to change HDF5 to use a new filter. Also all
HDF5 tools can work with all filters.
This is the way python works and how query languages execute user
defined functions.
I've used it myself successfully several times.
Cheers,
Ger
Mark Miller <miller86@llnl.gov> 10/13/2011 4:32 AM >>>
Sorry my last email was a bit off topic. I guess I latched on to the
"...we just need to know (in)formal requirements to integrate a new
filter" part.
So, related to this, I thought I recall a move afoot a couple of years
back now to formalize submission, repository of 3rd party HDF5
'filters'
of various sorts and compression in particular. What happened with
that?
I don't recall seeing much response on the forum to it but this sounds
like a good test case 
Mark
Hello all,
here at the DKRZ (German Climate Computing Center) we have to store
large volumes of climate data, some of which is stored in
HDF5-files.
So, during the last few month we have been doing some research into
climate data compression.
With very interesting results: We have been able to shrink our test
data
set to 38.76% of the original file size. This is a compression factor
of
more than 2.5 and it is significantly better than the performance of
all
the standard methods we tested (bzip2 = 54%, gzip = 58%, sldc = 66%
and
lzma = 46%). Also, we have seen that the lzma-algorithm performs
much
better than the other standard algorithms.
Even though we have constructed our methods to fit climate data, the
features we exploited for compression are very general and likely to
apply to other scientific data as well. This is why we are confident
that many of you could profit from these methods as well, and we
would
be happy to share our results with the rest of the community.
The filtering mechanism in HDF5 predestines it to be the first place
for
us to share our algorithms. But first we would be very interested to
see
the lzma algorithm integrated as an optional filtering method,
something
that should be very easy to do and offers large benefits to all
users.
Since we would be willing to do the necessary work, we just need to
know
the (in)formal requirements to integrate a new filter. And, of
course,
we would be very interested to hear about other recent work which
adresses compression in HDF5, and to get in touch with whoever works
on
···
On Tue, 2011-10-11 at 08:18 -0700, Nathanael Huebbe wrote:
it.
Best regards,
Nathanael Hübbe
http://wr.informatik.uni-hamburg.de/
http://wr.informatik.uni-hamburg.de/people/nathanael_huebbe
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org