Is there any guidance, rule of thumb, whatever on what compression level to use? I haven't been able to find much detail; apart from 0 being no compression and 9 being maximum. Is there a good balance in the range 1-9?
Is the selection as much about balancing processor vs. disk load as final size?
If I can speed writes with some lossless compression, that would be a good thing.
Scott
···
________________________________
This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail.
Quoting "Mitchell, Scott - IS" <Scott.Mitchell@itt.com>:
Is there any guidance, rule of thumb, whatever on what compression level to use? I haven't been able to find much detail; apart from 0 being no compression and 9 being maximum. Is there a good balance in the range 1-9?
Is the selection as much about balancing processor vs. disk load as final size?
If I can speed writes with some lossless compression, that would be a good thing.
Scott, the level to choose will probably depend on the data you're compressing.
You might also want to look at the LZF compression filter (http://h5py.alfven.org/lzf/). For the data I'm processing, it's been significantly faster than zlib.
-Steve
I have found level 1 zlib to be sufficient for cloud model floating point
data, and that CPU overhead becomes an issue as you approach 9... and you
get not much bang for the buck as you approach 9. If your data has a lot of
adjacent redundancy, level 1 will give you a lot of lossless compression at
only moderate cost.
Of course, the best way to answer your own question is to stick timings in
your code and try different levels and see what it gets you.
Leigh
···
On Fri, Mar 18, 2011 at 9:01 AM, Mitchell, Scott - IS < Scott.Mitchell@itt.com> wrote:
Is there any guidance, rule of thumb, whatever on what compression level
to use? I haven’t been able to find much detail; apart from 0 being no
compression and 9 being maximum. Is there a good balance in the range 1-9?
Is the selection as much about balancing processor vs. disk load as final
size?
If I can speed writes with some lossless compression, that would be a good
thing.
Scott
------------------------------
This e-mail and any files transmitted with it may be proprietary and are
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this e-mail in error please notify the
sender.
Please note that any views or opinions presented in this e-mail are solely
those of the author and do not necessarily represent those of ITT
Corporation. The recipient should check this e-mail and any attachments for
the presence of viruses. ITT accepts no liability for any damage caused by
any virus transmitted by this e-mail.
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Leigh Orf
Associate Professor of Atmospheric Science
Department of Geology and Meteorology
Central Michigan University
Currently on sabbatical at the National Center for Atmospheric Research
in Boulder, CO
NCAR office phone: (303) 497-8200
Is there an LZF filter for Windows 32 bit (and eventually 64 bit)?
Scott
···
-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-
bounces@hdfgroup.org] On Behalf Of stnchris@xmission.com
Sent: Friday, March 18, 2011 3:47 PM
To: hdf-forum@hdfgroup.org
Subject: Re: [Hdf-forum] Compression Level
Quoting "Mitchell, Scott - IS" <Scott.Mitchell@itt.com>:
> Is there any guidance, rule of thumb, whatever on what compression
> level to use? I haven't been able to find much detail; apart from 0
> being no compression and 9 being maximum. Is there a good balance in
> the range 1-9?
>
> Is the selection as much about balancing processor vs. disk load as
> final size?
>
> If I can speed writes with some lossless compression, that would be
> a good thing.
>
>
Scott, the level to choose will probably depend on the data you're
compressing.
You might also want to look at the LZF compression filter
(http://h5py.alfven.org/lzf/). For the data I'm processing, it's been
significantly faster than zlib.
-Steve
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail.
Yes, the LZF filter works fine on Windows (I'm using it on Windows XP, 32 and 64 bit). You'll need to download the h5py source tarball, grab the lzf filter and compression sources, and build it into your application (or standalone library).
There's a README file with the LZF filter that describes how to register the filter with HDF5.
-Steve
Quoting "Mitchell, Scott - IS" <Scott.Mitchell@itt.com>:
···
Is there an LZF filter for Windows 32 bit (and eventually 64 bit)?
Scott
-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-
bounces@hdfgroup.org] On Behalf Of stnchris@xmission.com
Sent: Friday, March 18, 2011 3:47 PM
To: hdf-forum@hdfgroup.org
Subject: Re: [Hdf-forum] Compression Level
Quoting "Mitchell, Scott - IS" <Scott.Mitchell@itt.com>:
> Is there any guidance, rule of thumb, whatever on what compression
> level to use? I haven't been able to find much detail; apart from 0
> being no compression and 9 being maximum. Is there a good balance in
> the range 1-9?
>
> Is the selection as much about balancing processor vs. disk load as
> final size?
>
> If I can speed writes with some lossless compression, that would be
> a good thing.
>
Scott, the level to choose will probably depend on the data you're
compressing.
You might also want to look at the LZF compression filter
(http://h5py.alfven.org/lzf/). For the data I'm processing, it's been
significantly faster than zlib.
-Steve
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail.
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
