# compression in hdf5 **URL:** \ **Category:** Hdf-forum archives\ **Created:** [January 29, 2008, 12:58am UTC](https://forum.hdfgroup.org/t/compression-in-hdf5/943 "2008-01-29T00:58:12Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![abhilash\_chandy](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/a/f0a364/32.png) [@abhilash\_chandy](https://forum.hdfgroup.org/u/abhilash_chandy)\ **Post date:** [January 29, 2008, 12:58am UTC](https://forum.hdfgroup.org/t/compression-in-hdf5/943/1 "2008-01-29T00:58:12Z")
Hi,   I am trying to test compression in hdf5 by testing a small program which just writes out one variable after creating a dataset. It kind of looks like this call h5fcreate\_f(filename, H5F\_ACC\_TRUNC\_F, file\_id, error) call h5screate\_simple\_f(rank,dim\_sizes,space\_id,hstatus) call h5pcreate\_f(H5P\_DATASET\_CREATE\_F, plist\_id, hstatus) call h5pset\_chunk\_f(plist\_id,rank,chunkdim\_sizes,hstatus) !Set the compression parameters comp\_prm(1)=6 ! not the best, but close (and fast) call h5pset\_deflate\_f(plist\_id, comp\_prm(1), hstatus) call h5tcopy\_f(H5T\_NATIVE\_REAL,type\_id,hstatus) call h5dcreate\_f(file\_id,vname,type\_id,space\_id,sds\_id,hstatus,plist\_id) call h5dwrite\_f(sds\_id,type\_id,variable,dim\_sizes,hstatus) Ofourse this is in hdf5. I try the same thing in hdf4 and the sample code looks like this sd\_id = sfstart(filename,DFACC\_CREATE) sds\_id = sfcreate(sd\_id,vname,DFNT\_FLOAT32, rank, dim\_sizes) status = sfscompress(sds\_id,COMP\_CODE\_DEFLATE,6) status = sfwdata(sds\_id, hstart, hstride, hedges, variable) status = sfendacc(sds\_id) status = sfend(sd\_id) Both the codes worked fine. The only problem was that I got an hdf5 file which was at least 5 times the size of the hdf4 file. That doesnt make sense. It should be the same right? No matter what value I gave for chunkdim\_sizes, the result was the same. Can anyone please help Thanks, Abhilash Chandy. > **···** > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > Be a better friend, newshound, and > know-it-all with Yahoo! Mobile. Try it now. [http://mobile.yahoo.com/;\_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > ---------------------------------------------------------------------- > This mailing list is for HDF software users discussion. > To subscribe to this list, send a message to [hdf-forum-subscribe@hdfgroup.org](mailto:hdf-forum-subscribe@hdfgroup.org). > To unsubscribe, send a message to [hdf-forum-unsubscribe@hdfgroup.org](mailto:hdf-forum-unsubscribe@hdfgroup.org). ---
**Author:** ![epourmal](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/e/c67d28/32.png) [@epourmal](https://forum.hdfgroup.org/u/epourmal)\ **Post date:** [January 29, 2008, 1:07am UTC](https://forum.hdfgroup.org/t/compression-in-hdf5/943/2 "2008-01-29T01:07:12Z")
Abhilash, If you data is small, size of the HDF5 overhead may be comparable with the size of the compressed data. Could you please send me the output of the h5dump -p -H \ ? Thank you! Elena > **···** > > At 4:58 PM -0800 1/28/08, abhilash chandy wrote: > > > Hi, > >   I am trying to test compression in hdf5 by testing a small program which just writes out one variable after creating a dataset. It kind of looks like this > > > > call h5fcreate\_f(filename, H5F\_ACC\_TRUNC\_F, file\_id, error) > > call h5screate\_simple\_f(rank,dim\_sizes,space\_id,hstatus) > > call h5pcreate\_f(H5P\_DATASET\_CREATE\_F, plist\_id, hstatus) > > call h5pset\_chunk\_f(plist\_id,rank,chunkdim\_sizes,hstatus) > > > > !Set the compression parameters > > comp\_prm(1)=6 ! not the best, but close (and fast) > > call h5pset\_deflate\_f(plist\_id, comp\_prm(1), hstatus) > > call h5tcopy\_f(H5T\_NATIVE\_REAL,type\_id,hstatus) > > call h5dcreate\_f(file\_id,vname,type\_id,space\_id,sds\_id,hstatus,plist\_id) > > call h5dwrite\_f(sds\_id,type\_id,variable,dim\_sizes,hstatus) > > > > Ofourse this is in hdf5. I try the same thing in hdf4 and the sample code looks like this > > > > sd\_id = sfstart(filename,DFACC\_CREATE) > > sds\_id = sfcreate(sd\_id,vname,DFNT\_FLOAT32, rank, dim\_sizes) > > status = sfscompress(sds\_id,COMP\_CODE\_DEFLATE,6) > > status = sfwdata(sds\_id, hstart, hstride, hedges, variable) > > status = sfendacc(sds\_id) > > status = sfend(sd\_id) > > > > Both the codes worked fine. The only problem was that I got an hdf5 file which was at least 5 times the size of the hdf4 file. That doesnt make sense. It should be the same right? No matter what value I gave for chunkdim\_sizes, the result was the same. > > > > Can anyone please help > > > > Thanks, > > Abhilash Chandy. > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Be a better friend, newshound, and > > know-it-all with Yahoo! Mobile. Try it now. [Home | Yahoo Mobile](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > > > ---------------------------------------------------------------------- > > This mailing list is for HDF software users discussion. > > To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org. > > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. > > -- > > ------------------------------------------------------------ > Elena Pourmal > The HDF Group > 1901 So First ST. > Suite C-2 > Champaign, IL 61820 > > epourmal@hdfgroup.org > (217)333-0238 (office) > (217)333-9049 (fax) > ------------------------------------------------------------ > > ---------------------------------------------------------------------- > This mailing list is for HDF software users discussion. > To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org. > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. ---
**Author:** ![abhilash\_chandy](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/a/f0a364/32.png) [@abhilash\_chandy](https://forum.hdfgroup.org/u/abhilash_chandy)\ **Post date:** [January 29, 2008, 12:31pm UTC](https://forum.hdfgroup.org/t/compression-in-hdf5/943/3 "2008-01-29T12:31:45Z")
Elena,   Okay. So I tried with chunk sizes 32x32x32. Actually I had tried them before (if you look at my first email, I tried with various chunk sizes) Here is the output of h5dump. Here I am getting a file of size 131072, while for hdf4, I am getting a size of 3209. That difference is huge. Not sure what is going on. HDF5 "var.hdf" { GROUP "/" {    DATASET "var" {       DATATYPE H5T\_IEEE\_F32LE       DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) }       STORAGE\_LAYOUT {          CHUNKED ( 32, 32, 32 )          SIZE 131072        }       FILTERS {          COMPRESSION SZIP {             PIXELS\_PER\_BLOCK 4             MODE K13             CODING NEAREST NEIGHBOUR                         HEADER RAW          }       }       FILLVALUE {          FILL\_TIME H5D\_FILL\_TIME\_IFSET          VALUE 0       }       ALLOCATION\_TIME {          H5D\_ALLOC\_TIME\_INCR       }    } } } > **···** > > ----- Original Message ---- > From: Elena Pourmal \ > To: abhilash chandy \; hdf-forum@hdfgroup.org > Sent: Monday, January 28, 2008 11:23:26 PM > Subject: Re: compression in hdf5 > > Please try chunk size 32x32x32 > > HDF5 applies compression to each chunk which is in this case 1 > element (4 bytes), no compression is applied at all. > > Elena > > At 6:46 PM -0800 1/28/08, abhilash chandy wrote: > > > Thanks for the reply. Here is the output of h5dump > > > > HDF5 "var.hdf" { > > GROUP "/" { > >    DATASET "var" { > >       DATATYPE H5T\_IEEE\_F32LE > >       DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) } > >       STORAGE\_LAYOUT { > >          CHUNKED ( 1, 1, 1 ) > >          SIZE 131072 > >        } > >       FILTERS { > >          COMPRESSION SZIP { > >             PIXELS\_PER\_BLOCK 4 > >             MODE K13 > >             CODING NEAREST NEIGHBOUR > >                         HEADER RAW > >          } > >       } > >       FILLVALUE { > >          FILL\_TIME H5D\_FILL\_TIME\_IFSET > >          VALUE 0 > >       } > >       ALLOCATION\_TIME { > >          H5D\_ALLOC\_TIME\_INCR > >       } > >    } > > } > > } > > > > ----- Original Message ---- > > From: Elena Pourmal \ > > To: abhilash chandy \; > > hdf-forum@hdfgroup.org > > > Sent: Monday, January 28, 2008 8:07:12 PM > > Subject: Re: compression in hdf5 > > > > Abhilash, > > > > If you data is small, size of the HDF5 overhead may be comparable > > with the size of the compressed data. > > > > Could you please send me the output of the > > > > h5dump -p -H \ > > ? > > > > Thank you! > > > > Elena > > > > At 4:58 PM -0800 1/28/08, abhilash chandy wrote: > > > > > Hi, > > >    I am trying to test compression in hdf5 by testing a small > > program > > > > which just writes out one variable after creating a dataset. It kind > > > of looks like this > > > > > > call h5fcreate\_f(filename, H5F\_ACC\_TRUNC\_F, file\_id, error) > > > call h5screate\_simple\_f(rank,dim\_sizes,space\_id,hstatus) > > > call h5pcreate\_f(H5P\_DATASET\_CREATE\_F, plist\_id, hstatus) > > > call h5pset\_chunk\_f(plist\_id,rank,chunkdim\_sizes,hstatus) > > > > > > !Set the compression parameters > > > comp\_prm(1)=6 ! not the best, but close (and fast) > > > call h5pset\_deflate\_f(plist\_id, comp\_prm(1), hstatus) > > > call h5tcopy\_f(H5T\_NATIVE\_REAL,type\_id,hstatus) > > > call > > > > h5dcreate\_f(file\_id,vname,type\_id,space\_id,sds\_id,hstatus,plist\_id) > > > > > call h5dwrite\_f(sds\_id,type\_id,variable,dim\_sizes,hstatus) > > > > > > Ofourse this is in hdf5. I try the same thing in hdf4 and the sample > > > code looks like this > > > > > > sd\_id = sfstart(filename,DFACC\_CREATE) > > > sds\_id = sfcreate(sd\_id,vname,DFNT\_FLOAT32, rank, dim\_sizes) > > > status = sfscompress(sds\_id,COMP\_CODE\_DEFLATE,6) > > > status = sfwdata(sds\_id, hstart, hstride, hedges, variable) > > > status = sfendacc(sds\_id) > > > status = sfend(sd\_id) > > > > > > Both the codes worked fine. The only problem was that I got an hdf5 > > > file which was at least 5 times the size of the hdf4 file. That > > > doesnt make sense. It should be the same right? No matter what value > > > I gave for chunkdim\_sizes, the result was the same. > > > > > > Can anyone please help > > > > > > Thanks, > > > Abhilash Chandy. > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > Be a better friend, newshound, and > > > know-it-all with Yahoo! Mobile. Try it now. > > > [Home | Yahoo Mobile](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > > > > > ---------------------------------------------------------------------- > > > This mailing list is for HDF software users discussion. > > > To subscribe to this list, send a message to > > > > hdf-forum-subscribe@hdfgroup.org. > > > > > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. > > > > -- > > > > ------------------------------------------------------------ > > Elena Pourmal > > The HDF Group > > 1901 So First ST. > > Suite C-2 > > Champaign, IL 61820 > > > > epourmal@hdfgroup.org > > (217)333-0238 (office) > > (217)333-9049 (fax) > > ------------------------------------------------------------ > > > > ---------------------------------------------------------------------- > > This mailing list is for HDF software users discussion. > > To subscribe to this list, send a message to > > hdf-forum-subscribe@hdfgroup.org. > > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Be a better friend, newshound, and > > know-it-all with Yahoo! Mobile. Try it now. > > [Home | Yahoo Mobile](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > -- > > ------------------------------------------------------------ > Elena Pourmal > The HDF Group > 1901 So First ST. > Suite C-2 > Champaign, IL 61820 > > epourmal@hdfgroup.org > (217)333-0238 (office) > (217)333-9049 (fax) > ------------------------------------------------------------ > >       \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > Be a better friend, newshound, and > know-it-all with Yahoo! Mobile. Try it now. [Home | Yahoo Mobile](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > ---------------------------------------------------------------------- > This mailing list is for HDF software users discussion. > To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org. > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. ---
**Author:** ![epourmal](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/e/c67d28/32.png) [@epourmal](https://forum.hdfgroup.org/u/epourmal)\ **Post date:** [January 29, 2008, 3:08pm UTC](https://forum.hdfgroup.org/t/compression-in-hdf5/943/4 "2008-01-29T15:08:07Z")
All, I am taking this discussion off line. There are may be several reasons why HDF5 is bigger. Abhilash and I will investigate and we will post our findings. Thank you! Elena > **···** > > At 4:31 AM -0800 1/29/08, abhilash chandy wrote: > > > Elena, > >   Okay. So I tried with chunk sizes 32x32x32. Actually I had tried them before (if you look at my first email, I tried with various chunk sizes) > > > > Here is the output of h5dump. Here I am getting a file of size 131072, while for hdf4, I am getting a size of 3209. That difference is huge. Not sure what is going on. > > > > HDF5 "var.hdf" { > > GROUP "/" { > >    DATASET "var" { > >       DATATYPE H5T\_IEEE\_F32LE > >       DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) } > >       STORAGE\_LAYOUT { > >          CHUNKED ( 32, 32, 32 ) > >          SIZE 131072 > >        } > >       FILTERS { > >          COMPRESSION SZIP { > >             PIXELS\_PER\_BLOCK 4 > >             MODE K13 > >             CODING NEAREST NEIGHBOUR > >                         HEADER RAW > >          } > >       } > >       FILLVALUE { > >          FILL\_TIME H5D\_FILL\_TIME\_IFSET > >          VALUE 0 } > >       ALLOCATION\_TIME { > >          H5D\_ALLOC\_TIME\_INCR > >       } > >    } > > } > > > > ----- Original Message ---- > > From: Elena Pourmal \ > > To: abhilash chandy \; hdf-forum@hdfgroup.org > > Sent: Monday, January 28, 2008 11:23:26 PM > > Subject: Re: compression in hdf5 > > > > Please try chunk size 32x32x32 > > > > HDF5 applies compression to each chunk which is in this case 1 > > element (4 bytes), no compression is applied at all. > > > > Elena > > > > At 6:46 PM -0800 1/28/08, abhilash chandy wrote: > > > > > Thanks for the reply. Here is the output of h5dump > > > > > > HDF5 "var.hdf" { > > > GROUP "/" { > > >     DATASET "var" { > > >        DATATYPE H5T\_IEEE\_F32LE > > >        DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) } > > >        STORAGE\_LAYOUT { > > >           CHUNKED ( 1, 1, 1 ) > > >           SIZE 131072 > > >         } > > >        FILTERS { > > >           COMPRESSION SZIP { > > >              PIXELS\_PER\_BLOCK 4 > > >              MODE K13 > > >              CODING NEAREST NEIGHBOUR > > >                          HEADER RAW > > >           } > > >        } > > >        FILLVALUE { > > >           FILL\_TIME H5D\_FILL\_TIME\_IFSET > > >           VALUE 0 } > > >        ALLOCATION\_TIME { > > >           H5D\_ALLOC\_TIME\_INCR > > >        } > > >     } > > > } > > > > > > ----- Original Message ---- > > > From: Elena Pourmal \ > > > To: abhilash chandy \; > > > > hdf-forum@hdfgroup.org > > > > > Sent: Monday, January 28, 2008 8:07:12 PM > > > Subject: Re: compression in hdf5 > > > > > > Abhilash, > > > > > > If you data is small, size of the HDF5 overhead may be comparable > > > with the size of the compressed data. > > > > > > Could you please send me the output of the > > > > > > h5dump -p -H \ > > > ? > > > > > > Thank you! > > > > > > Elena > > > > > > At 4:58 PM -0800 1/28/08, abhilash chandy wrote: > > > > > > > Hi, > > > >     I am trying to test compression in hdf5 by testing a small > > > > program > > > > > > which just writes out one variable after creating a dataset. It kind > > > > of looks like this > > > > > > > > call h5fcreate\_f(filename, H5F\_ACC\_TRUNC\_F, file\_id, error) > > > > call h5screate\_simple\_f(rank,dim\_sizes,space\_id,hstatus) > > > > call h5pcreate\_f(H5P\_DATASET\_CREATE\_F, plist\_id, hstatus) > > > > call h5pset\_chunk\_f(plist\_id,rank,chunkdim\_sizes,hstatus) > > > > > > > > !Set the compression parameters > > > > comp\_prm(1)=6 ! not the best, but close (and fast) > > > > call h5pset\_deflate\_f(plist\_id, comp\_prm(1), hstatus) > > > > call h5tcopy\_f(H5T\_NATIVE\_REAL,type\_id,hstatus) > > > > call > > > > > >   h5dcreate\_f(file\_id,vname,type\_id,space\_id,sds\_id,hstatus,plist\_id) > > > > > > > call h5dwrite\_f(sds\_id,type\_id,variable,dim\_sizes,hstatus) > > > > > > > > Ofourse this is in hdf5. I try the same thing in hdf4 and the sample > > > > code looks like this > > > > > > > > sd\_id = sfstart(filename,DFACC\_CREATE) > > > > sds\_id = sfcreate(sd\_id,vname,DFNT\_FLOAT32, rank, dim\_sizes) > > > > status = sfscompress(sds\_id,COMP\_CODE\_DEFLATE,6) > > > > status = sfwdata(sds\_id, hstart, hstride, hedges, variable) > > > > status = sfendacc(sds\_id) > > > > status = sfend(sd\_id) > > > > > > > > Both the codes worked fine. The only problem was that I got an hdf5 > > > > \>\>file which was at least 5 times the size of the hdf4 file. That > > > > > > doesnt make sense. It should be the same right? No matter what value > > > > I gave for chunkdim\_sizes, the result was the same. > > > > > > > > Can anyone please help > > > > > > > > Thanks, > > > > Abhilash Chandy. > > > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > > Be a better friend, newshound, and > > > > know-it-all with Yahoo! Mobile. Try it now. > > > > [Home | Yahoo Mobile](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > > > > > > > ---------------------------------------------------------------------- > > > > This mailing list is for HDF software users discussion. > > > > To subscribe to this list, send a message to > > > > > >   hdf-forum-subscribe@hdfgroup.org. > > > > > > > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. > > > > > > -- > > > > > > ------------------------------------------------------------ > > > Elena Pourmal > > > The HDF Group > > > 1901 So First ST. > > > Suite C-2 > > > Champaign, IL 61820 > > > > > > epourmal@hdfgroup.org > > > (217)333-0238 (office) > > > (217)333-9049 (fax) > > > ------------------------------------------------------------ > > > > > > ---------------------------------------------------------------------- > > > This mailing list is for HDF software users discussion. > > > To subscribe to this list, send a message to > > >   hdf-forum-subscribe@hdfgroup.org. > > > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > Be a better friend, newshound, and > > > know-it-all with Yahoo! Mobile. Try it now. > > > [Home | Yahoo Mobile](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > > > -- > > > > ------------------------------------------------------------ > > Elena Pourmal > > The HDF Group > > 1901 So First ST. > > Suite C-2 > > Champaign, IL 61820 > > > > epourmal@hdfgroup.org > > (217)333-0238 (office) > > (217)333-9049 (fax) > > ------------------------------------------------------------ > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Be a better friend, newshound, and > > know-it-all with Yahoo! Mobile. Try it now. [Home | Yahoo Mobile](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > -- > > ------------------------------------------------------------ > Elena Pourmal > The HDF Group > 1901 So First ST. > Suite C-2 > Champaign, IL 61820 > > epourmal@hdfgroup.org > (217)333-0238 (office) > (217)333-9049 (fax) > ------------------------------------------------------------ > > ---------------------------------------------------------------------- > This mailing list is for HDF software users discussion. > To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org. > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. ---
**Author:** ![abhilash\_chandy](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/a/f0a364/32.png) [@abhilash\_chandy](https://forum.hdfgroup.org/u/abhilash_chandy)\ **Post date:** [January 29, 2008, 2:46am UTC](https://forum.hdfgroup.org/t/compression-in-hdf5/943/5 "2008-01-29T02:46:10Z")
Thanks for the reply. Here is the output of h5dump HDF5 "var.hdf" { GROUP "/" {    DATASET "var" {       DATATYPE H5T\_IEEE\_F32LE       DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) }       STORAGE\_LAYOUT {          CHUNKED ( 1, 1, 1 )          SIZE 131072        }       FILTERS {          COMPRESSION SZIP {             PIXELS\_PER\_BLOCK 4             MODE K13             CODING NEAREST NEIGHBOUR                         HEADER RAW          }       }       FILLVALUE {          FILL\_TIME H5D\_FILL\_TIME\_IFSET          VALUE 0       }       ALLOCATION\_TIME {          H5D\_ALLOC\_TIME\_INCR       }    } } } > **···** > > ----- Original Message ---- > From: Elena Pourmal \ > To: abhilash chandy \; hdf-forum@hdfgroup.org > Sent: Monday, January 28, 2008 8:07:12 PM > Subject: Re: compression in hdf5 > > Abhilash, > > If you data is small, size of the HDF5 overhead may be comparable > with the size of the compressed data. > > Could you please send me the output of the > > h5dump -p -H \ > ? > > Thank you! > > Elena > > At 4:58 PM -0800 1/28/08, abhilash chandy wrote: > > > Hi, > >   I am trying to test compression in hdf5 by testing a small program > > which just writes out one variable after creating a dataset. It kind > > of looks like this > > > > call h5fcreate\_f(filename, H5F\_ACC\_TRUNC\_F, file\_id, error) > > call h5screate\_simple\_f(rank,dim\_sizes,space\_id,hstatus) > > call h5pcreate\_f(H5P\_DATASET\_CREATE\_F, plist\_id, hstatus) > > call h5pset\_chunk\_f(plist\_id,rank,chunkdim\_sizes,hstatus) > > > > !Set the compression parameters > > comp\_prm(1)=6 ! not the best, but close (and fast) > > call h5pset\_deflate\_f(plist\_id, comp\_prm(1), hstatus) > > call h5tcopy\_f(H5T\_NATIVE\_REAL,type\_id,hstatus) > > call > > h5dcreate\_f(file\_id,vname,type\_id,space\_id,sds\_id,hstatus,plist\_id) > > > call h5dwrite\_f(sds\_id,type\_id,variable,dim\_sizes,hstatus) > > > > Ofourse this is in hdf5. I try the same thing in hdf4 and the sample > > code looks like this > > > > sd\_id = sfstart(filename,DFACC\_CREATE) > > sds\_id = sfcreate(sd\_id,vname,DFNT\_FLOAT32, rank, dim\_sizes) > > status = sfscompress(sds\_id,COMP\_CODE\_DEFLATE,6) > > status = sfwdata(sds\_id, hstart, hstride, hedges, variable) > > status = sfendacc(sds\_id) > > status = sfend(sd\_id) > > > > Both the codes worked fine. The only problem was that I got an hdf5 > > file which was at least 5 times the size of the hdf4 file. That > > doesnt make sense. It should be the same right? No matter what value > > I gave for chunkdim\_sizes, the result was the same. > > > > Can anyone please help > > > > Thanks, > > Abhilash Chandy. > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Be a better friend, newshound, and > > know-it-all with Yahoo! Mobile. Try it now. > > [Home | Yahoo Mobile](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > > > ---------------------------------------------------------------------- > > This mailing list is for HDF software users discussion. > > To subscribe to this list, send a message to > > hdf-forum-subscribe@hdfgroup.org. > > > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. > > -- > > ------------------------------------------------------------ > Elena Pourmal > The HDF Group > 1901 So First ST. > Suite C-2 > Champaign, IL 61820 > > epourmal@hdfgroup.org > (217)333-0238 (office) > (217)333-9049 (fax) > ------------------------------------------------------------ > > ---------------------------------------------------------------------- > This mailing list is for HDF software users discussion. > To subscribe to this list, send a message to > hdf-forum-subscribe@hdfgroup.org. > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. > >       \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > Be a better friend, newshound, and > know-it-all with Yahoo! Mobile. Try it now. [Home | Yahoo Mobile](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > ---------------------------------------------------------------------- > This mailing list is for HDF software users discussion. > To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org. > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. ---
**Author:** ![Quincey\_Koziol](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/q/dbc845/32.png) [@Quincey\_Koziol](https://forum.hdfgroup.org/u/Quincey_Koziol)\ **Post date:** [January 29, 2008, 4:09am UTC](https://forum.hdfgroup.org/t/compression-in-hdf5/943/6 "2008-01-29T04:09:18Z")
Hi Abhilash, > Thanks for the reply. Here is the output of h5dump   Setting the size of the chunks to 1x1x1 is almost certainly not a good idea since the metadata to track each chunk will be [much] larger than the dataset elements themselves. You should increase the size of the chunks for your dataset to be close to what your I/O access on them will be. So, if you are planning on accessing the data in 8x8x8 blocks, make that your chunk size.   Quincey > **···** > > On Jan 28, 2008, at 8:46 PM, abhilash chandy wrote: > > > HDF5 "var.hdf" { > > GROUP "/" { > >   DATASET "var" { > >      DATATYPE H5T\_IEEE\_F32LE > >      DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) } > >      STORAGE\_LAYOUT { > >         CHUNKED ( 1, 1, 1 ) > >         SIZE 131072 > >       } > >      FILTERS { > >         COMPRESSION SZIP { > >            PIXELS\_PER\_BLOCK 4 > >            MODE K13 > >            CODING NEAREST NEIGHBOUR > >                        HEADER RAW > >         } > >      } > >      FILLVALUE { > >         FILL\_TIME H5D\_FILL\_TIME\_IFSET > >         VALUE 0 > >      } > >      ALLOCATION\_TIME { > >         H5D\_ALLOC\_TIME\_INCR > >      } > >   } > > } > > > > ----- Original Message ---- > > From: Elena Pourmal \ > > To: abhilash chandy \; hdf-forum@hdfgroup.org > > Sent: Monday, January 28, 2008 8:07:12 PM > > Subject: Re: compression in hdf5 > > > > Abhilash, > > > > If you data is small, size of the HDF5 overhead may be comparable > > with the size of the compressed data. > > > > Could you please send me the output of the > > > > h5dump -p -H \ > > ? > > > > Thank you! > > > > Elena > > > > At 4:58 PM -0800 1/28/08, abhilash chandy wrote: > > > > > Hi, > > > I am trying to test compression in hdf5 by testing a small program > > > which just writes out one variable after creating a dataset. It kind > > > of looks like this > > > > > > call h5fcreate\_f(filename, H5F\_ACC\_TRUNC\_F, file\_id, error) > > > call h5screate\_simple\_f(rank,dim\_sizes,space\_id,hstatus) > > > call h5pcreate\_f(H5P\_DATASET\_CREATE\_F, plist\_id, hstatus) > > > call h5pset\_chunk\_f(plist\_id,rank,chunkdim\_sizes,hstatus) > > > > > > !Set the compression parameters > > > comp\_prm(1)=6 ! not the best, but close (and fast) > > > call h5pset\_deflate\_f(plist\_id, comp\_prm(1), hstatus) > > > call h5tcopy\_f(H5T\_NATIVE\_REAL,type\_id,hstatus) > > > call > > > > h5dcreate\_f(file\_id,vname,type\_id,space\_id,sds\_id,hstatus,plist\_id) > > > > > call h5dwrite\_f(sds\_id,type\_id,variable,dim\_sizes,hstatus) > > > > > > Ofourse this is in hdf5. I try the same thing in hdf4 and the sample > > > code looks like this > > > > > > sd\_id = sfstart(filename,DFACC\_CREATE) > > > sds\_id = sfcreate(sd\_id,vname,DFNT\_FLOAT32, rank, dim\_sizes) > > > status = sfscompress(sds\_id,COMP\_CODE\_DEFLATE,6) > > > status = sfwdata(sds\_id, hstart, hstride, hedges, variable) > > > status = sfendacc(sds\_id) > > > status = sfend(sd\_id) > > > > > > Both the codes worked fine. The only problem was that I got an hdf5 > > > file which was at least 5 times the size of the hdf4 file. That > > > doesnt make sense. It should be the same right? No matter what value > > > I gave for chunkdim\_sizes, the result was the same. > > > > > > Can anyone please help > > > > > > Thanks, > > > Abhilash Chandy. > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > Be a better friend, newshound, and > > > know-it-all with Yahoo! Mobile. Try it now. > > > [Home | Yahoo Mobile](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > > > > > ---------------------------------------------------------------------- > > > This mailing list is for HDF software users discussion. > > > To subscribe to this list, send a message to > > > > hdf-forum-subscribe@hdfgroup.org. > > > > > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. > > > > -- > > > > ------------------------------------------------------------ > > Elena Pourmal > > The HDF Group > > 1901 So First ST. > > Suite C-2 > > Champaign, IL 61820 > > > > epourmal@hdfgroup.org > > (217)333-0238 (office) > > (217)333-9049 (fax) > > ------------------------------------------------------------ > > > > ---------------------------------------------------------------------- > > This mailing list is for HDF software users discussion. > > To subscribe to this list, send a message to > > hdf-forum-subscribe@hdfgroup.org. > > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. > > > >      \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Be a better friend, newshound, and > > know-it-all with Yahoo! Mobile. Try it now. [Home | Yahoo Mobile](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > > > ---------------------------------------------------------------------- > > This mailing list is for HDF software users discussion. > > To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org. > > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. > > ---------------------------------------------------------------------- > This mailing list is for HDF software users discussion. > To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org. > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. ---
**Author:** ![epourmal](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/e/c67d28/32.png) [@epourmal](https://forum.hdfgroup.org/u/epourmal)\ **Post date:** [January 29, 2008, 4:23am UTC](https://forum.hdfgroup.org/t/compression-in-hdf5/943/7 "2008-01-29T04:23:26Z")
Please try chunk size 32x32x32 HDF5 applies compression to each chunk which is in this case 1 element (4 bytes), no compression is applied at all. Elena > **···** > > At 6:46 PM -0800 1/28/08, abhilash chandy wrote: > > > Thanks for the reply. Here is the output of h5dump > > > > HDF5 "var.hdf" { > > GROUP "/" { > >    DATASET "var" { > >       DATATYPE H5T\_IEEE\_F32LE > >       DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) } > >       STORAGE\_LAYOUT { > >          CHUNKED ( 1, 1, 1 ) > >          SIZE 131072 > >        } > >       FILTERS { > >          COMPRESSION SZIP { > >             PIXELS\_PER\_BLOCK 4 > >             MODE K13 > >             CODING NEAREST NEIGHBOUR > >                         HEADER RAW > >          } > >       } > >       FILLVALUE { > >          FILL\_TIME H5D\_FILL\_TIME\_IFSET > >          VALUE 0 } > >       ALLOCATION\_TIME { > >          H5D\_ALLOC\_TIME\_INCR > >       } > >    } > > } > > > > ----- Original Message ---- > > From: Elena Pourmal \ > > To: abhilash chandy \; hdf-forum@hdfgroup.org > > Sent: Monday, January 28, 2008 8:07:12 PM > > Subject: Re: compression in hdf5 > > > > Abhilash, > > > > If you data is small, size of the HDF5 overhead may be comparable > > with the size of the compressed data. > > > > Could you please send me the output of the > > > > h5dump -p -H \ > > ? > > > > Thank you! > > > > Elena > > > > At 4:58 PM -0800 1/28/08, abhilash chandy wrote: > > > > > Hi, > > >    I am trying to test compression in hdf5 by testing a small program > > > which just writes out one variable after creating a dataset. It kind > > > of looks like this > > > > > > call h5fcreate\_f(filename, H5F\_ACC\_TRUNC\_F, file\_id, error) > > > call h5screate\_simple\_f(rank,dim\_sizes,space\_id,hstatus) > > > call h5pcreate\_f(H5P\_DATASET\_CREATE\_F, plist\_id, hstatus) > > > call h5pset\_chunk\_f(plist\_id,rank,chunkdim\_sizes,hstatus) > > > > > > !Set the compression parameters > > > comp\_prm(1)=6 ! not the best, but close (and fast) > > > call h5pset\_deflate\_f(plist\_id, comp\_prm(1), hstatus) > > > call h5tcopy\_f(H5T\_NATIVE\_REAL,type\_id,hstatus) > > > call > > > > h5dcreate\_f(file\_id,vname,type\_id,space\_id,sds\_id,hstatus,plist\_id) > > > > > call h5dwrite\_f(sds\_id,type\_id,variable,dim\_sizes,hstatus) > > > > > > Ofourse this is in hdf5. I try the same thing in hdf4 and the sample > > > code looks like this > > > > > > sd\_id = sfstart(filename,DFACC\_CREATE) > > > sds\_id = sfcreate(sd\_id,vname,DFNT\_FLOAT32, rank, dim\_sizes) > > > status = sfscompress(sds\_id,COMP\_CODE\_DEFLATE,6) > > > status = sfwdata(sds\_id, hstart, hstride, hedges, variable) > > > status = sfendacc(sds\_id) > > > status = sfend(sd\_id) > > > > > > Both the codes worked fine. The only problem was that I got an hdf5 > > > file which was at least 5 times the size of the hdf4 file. That > > > doesnt make sense. It should be the same right? No matter what value > > > I gave for chunkdim\_sizes, the result was the same. > > > > > > Can anyone please help > > > > > > Thanks, > > > Abhilash Chandy. > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > Be a better friend, newshound, and > > > know-it-all with Yahoo! Mobile. Try it now. > > > [Home | Yahoo Mobile](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > > > > > ---------------------------------------------------------------------- > > > This mailing list is for HDF software users discussion. > > > To subscribe to this list, send a message to > > > > hdf-forum-subscribe@hdfgroup.org. > > > > > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. > > > > -- > > > > ------------------------------------------------------------ > > Elena Pourmal > > The HDF Group > > 1901 So First ST. > > Suite C-2 > > Champaign, IL 61820 > > > > epourmal@hdfgroup.org > > (217)333-0238 (office) > > (217)333-9049 (fax) > > ------------------------------------------------------------ > > > > ---------------------------------------------------------------------- > > This mailing list is for HDF software users discussion. > > To subscribe to this list, send a message to > > hdf-forum-subscribe@hdfgroup.org. > > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org. > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Be a better friend, newshound, and > > know-it-all with Yahoo! Mobile. Try it now. [Home | Yahoo Mobile](http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ) > > -- > > ------------------------------------------------------------ > Elena Pourmal > The HDF Group > 1901 So First ST. > Suite C-2 > Champaign, IL 61820 > > epourmal@hdfgroup.org > (217)333-0238 (office) > (217)333-9049 (fax) > ------------------------------------------------------------ > > ---------------------------------------------------------------------- > This mailing list is for HDF software users discussion. > To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org. > To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.