gzip compression

Hdf-forum archives
1

Hi,

I think this question has been already asked, but I can not find a clear
conclusion.

I want to mimic the gzip compression level (0-9), e.g., as done using
h5repack -f GZIP=N.

What does h5repack takes as a compressed chunk size? What should I pass
to "H5Pset_chunk(plist, rank, cdims);" in my C code? Currently I do

for(int j=0; j<rank; j++) cdims[j] = std::min(dims[j],hsize_t(1e6));

but I am worried if that is the right thing to do.

Thanks and regards,
Dominik

2

h5repack uses the total chunks size of H5TOOLS_BUFSIZE, which is 1MB,
allocated from the the fastest dimension first.

Your chunk size may run into problem if you have 4D+ and each dim size is
1e5 for instance.

There are a couple of forum threads on this issue. You may need to take a look
how others use chunk size.

Thanks
--pc

Dominik Szczerba wrote:

···

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

I think this question has been already asked, but I can not find a clear
conclusion.

I want to mimic the gzip compression level (0-9), e.g., as done using
h5repack -f GZIP=N.

What does h5repack takes as a compressed chunk size? What should I pass
to "H5Pset_chunk(plist, rank, cdims);" in my C code? Currently I do

for(int j=0; j<rank; j++) cdims[j] = std::min(dims[j],hsize_t(1e6));

but I am worried if that is the right thing to do.

Thanks and regards,
Dominik
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAktm+SUACgkQ/EBMh9bUuzLUAACgohsYaru7CJ1eoOJPBwP3ZWZZ
dDQAn1sYPKxQ4Nxx/ruiLaHy23XI+saN
=Oser
-----END PGP SIGNATURE-----

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

3

Hi Peter,

I am just happy to do what h5repack does for the moment, so I only added
missing division by element size. Interestingly, I have unaware chosen
the same bufsize of 1e6.

Thanks,
Dominik

Peter Cao wrote:

···

h5repack uses the total chunks size of H5TOOLS_BUFSIZE, which is 1MB,
allocated from the the fastest dimension first.

Your chunk size may run into problem if you have 4D+ and each dim size is
1e5 for instance.

There are a couple of forum threads on this issue. You may need to take
a look
how others use chunk size.

Thanks
--pc

Dominik Szczerba wrote:
Hi,

I think this question has been already asked, but I can not find a clear
conclusion.

I want to mimic the gzip compression level (0-9), e.g., as done using
h5repack -f GZIP=N.

What does h5repack takes as a compressed chunk size? What should I pass
to "H5Pset_chunk(plist, rank, cdims);" in my C code? Currently I do

for(int j=0; j<rank; j++) cdims[j] = std::min(dims[j],hsize_t(1e6));

but I am worried if that is the right thing to do.

Thanks and regards,
Dominik

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org