compression in hdf5

Hdf-forum archives
1

Hi,
  I am trying to test compression in hdf5 by testing a small program which just writes out one variable after creating a dataset. It kind of looks like this

call h5fcreate_f(filename, H5F_ACC_TRUNC_F, file_id, error)
call h5screate_simple_f(rank,dim_sizes,space_id,hstatus)
call h5pcreate_f(H5P_DATASET_CREATE_F, plist_id, hstatus)
call h5pset_chunk_f(plist_id,rank,chunkdim_sizes,hstatus)

!Set the compression parameters
comp_prm(1)=6 ! not the best, but close (and fast)
call h5pset_deflate_f(plist_id, comp_prm(1), hstatus)
call h5tcopy_f(H5T_NATIVE_REAL,type_id,hstatus)
call h5dcreate_f(file_id,vname,type_id,space_id,sds_id,hstatus,plist_id)
call h5dwrite_f(sds_id,type_id,variable,dim_sizes,hstatus)

Ofourse this is in hdf5. I try the same thing in hdf4 and the sample code looks like this

sd_id = sfstart(filename,DFACC_CREATE)
sds_id = sfcreate(sd_id,vname,DFNT_FLOAT32, rank, dim_sizes)
status = sfscompress(sds_id,COMP_CODE_DEFLATE,6)
status = sfwdata(sds_id, hstart, hstride, hedges, variable)
status = sfendacc(sds_id)
status = sfend(sd_id)

Both the codes worked fine. The only problem was that I got an hdf5 file which was at least 5 times the size of the hdf4 file. That doesnt make sense. It should be the same right? No matter what value I gave for chunkdim_sizes, the result was the same.

Can anyone please help

Thanks,
Abhilash Chandy.

···

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

2

Abhilash,

If you data is small, size of the HDF5 overhead may be comparable with the size of the compressed data.

Could you please send me the output of the

h5dump -p -H <yourfile.h5>
?

Thank you!

Elena

···

At 4:58 PM -0800 1/28/08, abhilash chandy wrote:

Hi,
  I am trying to test compression in hdf5 by testing a small program which just writes out one variable after creating a dataset. It kind of looks like this

call h5fcreate_f(filename, H5F_ACC_TRUNC_F, file_id, error)
call h5screate_simple_f(rank,dim_sizes,space_id,hstatus)
call h5pcreate_f(H5P_DATASET_CREATE_F, plist_id, hstatus)
call h5pset_chunk_f(plist_id,rank,chunkdim_sizes,hstatus)

!Set the compression parameters
comp_prm(1)=6 ! not the best, but close (and fast)
call h5pset_deflate_f(plist_id, comp_prm(1), hstatus)
call h5tcopy_f(H5T_NATIVE_REAL,type_id,hstatus)
call h5dcreate_f(file_id,vname,type_id,space_id,sds_id,hstatus,plist_id)
call h5dwrite_f(sds_id,type_id,variable,dim_sizes,hstatus)

Ofourse this is in hdf5. I try the same thing in hdf4 and the sample code looks like this

sd_id = sfstart(filename,DFACC_CREATE)
sds_id = sfcreate(sd_id,vname,DFNT_FLOAT32, rank, dim_sizes)
status = sfscompress(sds_id,COMP_CODE_DEFLATE,6)
status = sfwdata(sds_id, hstart, hstride, hedges, variable)
status = sfendacc(sds_id)
status = sfend(sd_id)

Both the codes worked fine. The only problem was that I got an hdf5 file which was at least 5 times the size of the hdf4 file. That doesnt make sense. It should be the same right? No matter what value I gave for chunkdim_sizes, the result was the same.

Can anyone please help

Thanks,
Abhilash Chandy.

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

--

------------------------------------------------------------
Elena Pourmal
The HDF Group
1901 So First ST.
Suite C-2
Champaign, IL 61820

epourmal@hdfgroup.org
(217)333-0238 (office)
(217)333-9049 (fax)
------------------------------------------------------------

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

3

Elena,
  Okay. So I tried with chunk sizes 32x32x32. Actually I had tried them before (if you look at my first email, I tried with various chunk sizes)

Here is the output of h5dump. Here I am getting a file of size 131072, while for hdf4, I am getting a size of 3209. That difference is huge. Not sure what is going on.

HDF5 "var.hdf" {
GROUP "/" {
   DATASET "var" {
      DATATYPE H5T_IEEE_F32LE
      DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) }
      STORAGE_LAYOUT {
         CHUNKED ( 32, 32, 32 )
         SIZE 131072
       }
      FILTERS {
         COMPRESSION SZIP {
            PIXELS_PER_BLOCK 4
            MODE K13
            CODING NEAREST NEIGHBOUR
                        HEADER RAW
         }
      }
      FILLVALUE {
         FILL_TIME H5D_FILL_TIME_IFSET
         VALUE 0
      }
      ALLOCATION_TIME {
         H5D_ALLOC_TIME_INCR
      }
   }
}
}

···

----- Original Message ----
From: Elena Pourmal <epourmal@hdfgroup.org>
To: abhilash chandy <abhilash_chandy@yahoo.com>; hdf-forum@hdfgroup.org
Sent: Monday, January 28, 2008 11:23:26 PM
Subject: Re: compression in hdf5

Please try chunk size 32x32x32

HDF5 applies compression to each chunk which is in this case 1
element (4 bytes), no compression is applied at all.

Elena

At 6:46 PM -0800 1/28/08, abhilash chandy wrote:

Thanks for the reply. Here is the output of h5dump

HDF5 "var.hdf" {
GROUP "/" {
   DATASET "var" {
      DATATYPE H5T_IEEE_F32LE
      DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) }
      STORAGE_LAYOUT {
         CHUNKED ( 1, 1, 1 )
         SIZE 131072
       }
      FILTERS {
         COMPRESSION SZIP {
            PIXELS_PER_BLOCK 4
            MODE K13
            CODING NEAREST NEIGHBOUR
                        HEADER RAW
         }
      }
      FILLVALUE {
         FILL_TIME H5D_FILL_TIME_IFSET
         VALUE 0
      }
      ALLOCATION_TIME {
         H5D_ALLOC_TIME_INCR
      }
   }
}
}

----- Original Message ----
From: Elena Pourmal <epourmal@hdfgroup.org>
To: abhilash chandy <abhilash_chandy@yahoo.com>;

hdf-forum@hdfgroup.org

Sent: Monday, January 28, 2008 8:07:12 PM
Subject: Re: compression in hdf5

Abhilash,

If you data is small, size of the HDF5 overhead may be comparable
with the size of the compressed data.

Could you please send me the output of the

h5dump -p -H <yourfile.h5>
?

Thank you!

Elena

At 4:58 PM -0800 1/28/08, abhilash chandy wrote:

Hi,
   I am trying to test compression in hdf5 by testing a small

program

which just writes out one variable after creating a dataset. It kind
of looks like this

call h5fcreate_f(filename, H5F_ACC_TRUNC_F, file_id, error)
call h5screate_simple_f(rank,dim_sizes,space_id,hstatus)
call h5pcreate_f(H5P_DATASET_CREATE_F, plist_id, hstatus)
call h5pset_chunk_f(plist_id,rank,chunkdim_sizes,hstatus)

!Set the compression parameters
comp_prm(1)=6 ! not the best, but close (and fast)
call h5pset_deflate_f(plist_id, comp_prm(1), hstatus)
call h5tcopy_f(H5T_NATIVE_REAL,type_id,hstatus)
call

h5dcreate_f(file_id,vname,type_id,space_id,sds_id,hstatus,plist_id)

call h5dwrite_f(sds_id,type_id,variable,dim_sizes,hstatus)

Ofourse this is in hdf5. I try the same thing in hdf4 and the sample
code looks like this

sd_id = sfstart(filename,DFACC_CREATE)
sds_id = sfcreate(sd_id,vname,DFNT_FLOAT32, rank, dim_sizes)
status = sfscompress(sds_id,COMP_CODE_DEFLATE,6)
status = sfwdata(sds_id, hstart, hstride, hedges, variable)
status = sfendacc(sds_id)
status = sfend(sd_id)

Both the codes worked fine. The only problem was that I got an hdf5
file which was at least 5 times the size of the hdf4 file. That
doesnt make sense. It should be the same right? No matter what value
I gave for chunkdim_sizes, the result was the same.

Can anyone please help

Thanks,
Abhilash Chandy.

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to

hdf-forum-subscribe@hdfgroup.org.

To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

--

------------------------------------------------------------
Elena Pourmal
The HDF Group
1901 So First ST.
Suite C-2
Champaign, IL 61820

epourmal@hdfgroup.org
(217)333-0238 (office)
(217)333-9049 (fax)
------------------------------------------------------------

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

--

------------------------------------------------------------
Elena Pourmal
The HDF Group
1901 So First ST.
Suite C-2
Champaign, IL 61820

epourmal@hdfgroup.org
(217)333-0238 (office)
(217)333-9049 (fax)
------------------------------------------------------------

      ____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

4

All,

I am taking this discussion off line. There are may be several reasons why HDF5 is bigger.
Abhilash and I will investigate and we will post our findings.

Thank you!

Elena

···

At 4:31 AM -0800 1/29/08, abhilash chandy wrote:

Elena,
  Okay. So I tried with chunk sizes 32x32x32. Actually I had tried them before (if you look at my first email, I tried with various chunk sizes)

Here is the output of h5dump. Here I am getting a file of size 131072, while for hdf4, I am getting a size of 3209. That difference is huge. Not sure what is going on.

HDF5 "var.hdf" {
GROUP "/" {
   DATASET "var" {
      DATATYPE H5T_IEEE_F32LE
      DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) }
      STORAGE_LAYOUT {
         CHUNKED ( 32, 32, 32 )
         SIZE 131072
       }
      FILTERS {
         COMPRESSION SZIP {
            PIXELS_PER_BLOCK 4
            MODE K13
            CODING NEAREST NEIGHBOUR
                        HEADER RAW
         }
      }
      FILLVALUE {
         FILL_TIME H5D_FILL_TIME_IFSET
         VALUE 0 }
      ALLOCATION_TIME {
         H5D_ALLOC_TIME_INCR
      }
   }
}

----- Original Message ----
From: Elena Pourmal <epourmal@hdfgroup.org>
To: abhilash chandy <abhilash_chandy@yahoo.com>; hdf-forum@hdfgroup.org
Sent: Monday, January 28, 2008 11:23:26 PM
Subject: Re: compression in hdf5

Please try chunk size 32x32x32

HDF5 applies compression to each chunk which is in this case 1
element (4 bytes), no compression is applied at all.

Elena

At 6:46 PM -0800 1/28/08, abhilash chandy wrote:

Thanks for the reply. Here is the output of h5dump

HDF5 "var.hdf" {
GROUP "/" {
    DATASET "var" {
       DATATYPE H5T_IEEE_F32LE
       DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) }
       STORAGE_LAYOUT {
          CHUNKED ( 1, 1, 1 )
          SIZE 131072
        }
       FILTERS {
          COMPRESSION SZIP {
             PIXELS_PER_BLOCK 4
             MODE K13
             CODING NEAREST NEIGHBOUR
                         HEADER RAW
          }
       }
       FILLVALUE {
          FILL_TIME H5D_FILL_TIME_IFSET
          VALUE 0 }
       ALLOCATION_TIME {
          H5D_ALLOC_TIME_INCR
       }
    }
}

----- Original Message ----
From: Elena Pourmal <epourmal@hdfgroup.org>
To: abhilash chandy <abhilash_chandy@yahoo.com>;

hdf-forum@hdfgroup.org

Sent: Monday, January 28, 2008 8:07:12 PM
Subject: Re: compression in hdf5

Abhilash,

If you data is small, size of the HDF5 overhead may be comparable
with the size of the compressed data.

Could you please send me the output of the

h5dump -p -H <yourfile.h5>
?

Thank you!

Elena

At 4:58 PM -0800 1/28/08, abhilash chandy wrote:

Hi,
    I am trying to test compression in hdf5 by testing a small

program

which just writes out one variable after creating a dataset. It kind
of looks like this

call h5fcreate_f(filename, H5F_ACC_TRUNC_F, file_id, error)
call h5screate_simple_f(rank,dim_sizes,space_id,hstatus)
call h5pcreate_f(H5P_DATASET_CREATE_F, plist_id, hstatus)
call h5pset_chunk_f(plist_id,rank,chunkdim_sizes,hstatus)

!Set the compression parameters
comp_prm(1)=6 ! not the best, but close (and fast)
call h5pset_deflate_f(plist_id, comp_prm(1), hstatus)
call h5tcopy_f(H5T_NATIVE_REAL,type_id,hstatus)
call

  h5dcreate_f(file_id,vname,type_id,space_id,sds_id,hstatus,plist_id)

call h5dwrite_f(sds_id,type_id,variable,dim_sizes,hstatus)

Ofourse this is in hdf5. I try the same thing in hdf4 and the sample
code looks like this

sd_id = sfstart(filename,DFACC_CREATE)
sds_id = sfcreate(sd_id,vname,DFNT_FLOAT32, rank, dim_sizes)
status = sfscompress(sds_id,COMP_CODE_DEFLATE,6)
status = sfwdata(sds_id, hstart, hstride, hedges, variable)
status = sfendacc(sds_id)
status = sfend(sd_id)

Both the codes worked fine. The only problem was that I got an hdf5

>>file which was at least 5 times the size of the hdf4 file. That

doesnt make sense. It should be the same right? No matter what value
I gave for chunkdim_sizes, the result was the same.

Can anyone please help

Thanks,
Abhilash Chandy.

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to

  hdf-forum-subscribe@hdfgroup.org.

To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

--

------------------------------------------------------------
Elena Pourmal
The HDF Group
1901 So First ST.
Suite C-2
Champaign, IL 61820

epourmal@hdfgroup.org
(217)333-0238 (office)
(217)333-9049 (fax)
------------------------------------------------------------

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
  hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

--

------------------------------------------------------------
Elena Pourmal
The HDF Group
1901 So First ST.
Suite C-2
Champaign, IL 61820

epourmal@hdfgroup.org
(217)333-0238 (office)
(217)333-9049 (fax)
------------------------------------------------------------

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

--

------------------------------------------------------------
Elena Pourmal
The HDF Group
1901 So First ST.
Suite C-2
Champaign, IL 61820

epourmal@hdfgroup.org
(217)333-0238 (office)
(217)333-9049 (fax)
------------------------------------------------------------

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

5

Thanks for the reply. Here is the output of h5dump

HDF5 "var.hdf" {
GROUP "/" {
   DATASET "var" {
      DATATYPE H5T_IEEE_F32LE
      DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) }
      STORAGE_LAYOUT {
         CHUNKED ( 1, 1, 1 )
         SIZE 131072
       }
      FILTERS {
         COMPRESSION SZIP {
            PIXELS_PER_BLOCK 4
            MODE K13
            CODING NEAREST NEIGHBOUR
                        HEADER RAW
         }
      }
      FILLVALUE {
         FILL_TIME H5D_FILL_TIME_IFSET
         VALUE 0
      }
      ALLOCATION_TIME {
         H5D_ALLOC_TIME_INCR
      }
   }
}
}

···

----- Original Message ----
From: Elena Pourmal <epourmal@hdfgroup.org>
To: abhilash chandy <abhilash_chandy@yahoo.com>; hdf-forum@hdfgroup.org
Sent: Monday, January 28, 2008 8:07:12 PM
Subject: Re: compression in hdf5

Abhilash,

If you data is small, size of the HDF5 overhead may be comparable
with the size of the compressed data.

Could you please send me the output of the

h5dump -p -H <yourfile.h5>
?

Thank you!

Elena

At 4:58 PM -0800 1/28/08, abhilash chandy wrote:

Hi,
  I am trying to test compression in hdf5 by testing a small program
which just writes out one variable after creating a dataset. It kind
of looks like this

call h5fcreate_f(filename, H5F_ACC_TRUNC_F, file_id, error)
call h5screate_simple_f(rank,dim_sizes,space_id,hstatus)
call h5pcreate_f(H5P_DATASET_CREATE_F, plist_id, hstatus)
call h5pset_chunk_f(plist_id,rank,chunkdim_sizes,hstatus)

!Set the compression parameters
comp_prm(1)=6 ! not the best, but close (and fast)
call h5pset_deflate_f(plist_id, comp_prm(1), hstatus)
call h5tcopy_f(H5T_NATIVE_REAL,type_id,hstatus)
call

h5dcreate_f(file_id,vname,type_id,space_id,sds_id,hstatus,plist_id)

call h5dwrite_f(sds_id,type_id,variable,dim_sizes,hstatus)

Ofourse this is in hdf5. I try the same thing in hdf4 and the sample
code looks like this

sd_id = sfstart(filename,DFACC_CREATE)
sds_id = sfcreate(sd_id,vname,DFNT_FLOAT32, rank, dim_sizes)
status = sfscompress(sds_id,COMP_CODE_DEFLATE,6)
status = sfwdata(sds_id, hstart, hstride, hedges, variable)
status = sfendacc(sds_id)
status = sfend(sd_id)

Both the codes worked fine. The only problem was that I got an hdf5
file which was at least 5 times the size of the hdf4 file. That
doesnt make sense. It should be the same right? No matter what value
I gave for chunkdim_sizes, the result was the same.

Can anyone please help

Thanks,
Abhilash Chandy.

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to

hdf-forum-subscribe@hdfgroup.org.

To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

--

------------------------------------------------------------
Elena Pourmal
The HDF Group
1901 So First ST.
Suite C-2
Champaign, IL 61820

epourmal@hdfgroup.org
(217)333-0238 (office)
(217)333-9049 (fax)
------------------------------------------------------------

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

      ____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

6

Hi Abhilash,

Thanks for the reply. Here is the output of h5dump

  Setting the size of the chunks to 1x1x1 is almost certainly not a good idea since the metadata to track each chunk will be [much] larger than the dataset elements themselves. You should increase the size of the chunks for your dataset to be close to what your I/O access on them will be. So, if you are planning on accessing the data in 8x8x8 blocks, make that your chunk size.

  Quincey

···

On Jan 28, 2008, at 8:46 PM, abhilash chandy wrote:

HDF5 "var.hdf" {
GROUP "/" {
  DATASET "var" {
     DATATYPE H5T_IEEE_F32LE
     DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) }
     STORAGE_LAYOUT {
        CHUNKED ( 1, 1, 1 )
        SIZE 131072
      }
     FILTERS {
        COMPRESSION SZIP {
           PIXELS_PER_BLOCK 4
           MODE K13
           CODING NEAREST NEIGHBOUR
                       HEADER RAW
        }
     }
     FILLVALUE {
        FILL_TIME H5D_FILL_TIME_IFSET
        VALUE 0
     }
     ALLOCATION_TIME {
        H5D_ALLOC_TIME_INCR
     }
  }
}

----- Original Message ----
From: Elena Pourmal <epourmal@hdfgroup.org>
To: abhilash chandy <abhilash_chandy@yahoo.com>; hdf-forum@hdfgroup.org
Sent: Monday, January 28, 2008 8:07:12 PM
Subject: Re: compression in hdf5

Abhilash,

If you data is small, size of the HDF5 overhead may be comparable
with the size of the compressed data.

Could you please send me the output of the

h5dump -p -H <yourfile.h5>
?

Thank you!

Elena

At 4:58 PM -0800 1/28/08, abhilash chandy wrote:

Hi,
I am trying to test compression in hdf5 by testing a small program
which just writes out one variable after creating a dataset. It kind
of looks like this

call h5fcreate_f(filename, H5F_ACC_TRUNC_F, file_id, error)
call h5screate_simple_f(rank,dim_sizes,space_id,hstatus)
call h5pcreate_f(H5P_DATASET_CREATE_F, plist_id, hstatus)
call h5pset_chunk_f(plist_id,rank,chunkdim_sizes,hstatus)

!Set the compression parameters
comp_prm(1)=6 ! not the best, but close (and fast)
call h5pset_deflate_f(plist_id, comp_prm(1), hstatus)
call h5tcopy_f(H5T_NATIVE_REAL,type_id,hstatus)
call

h5dcreate_f(file_id,vname,type_id,space_id,sds_id,hstatus,plist_id)

call h5dwrite_f(sds_id,type_id,variable,dim_sizes,hstatus)

Ofourse this is in hdf5. I try the same thing in hdf4 and the sample
code looks like this

sd_id = sfstart(filename,DFACC_CREATE)
sds_id = sfcreate(sd_id,vname,DFNT_FLOAT32, rank, dim_sizes)
status = sfscompress(sds_id,COMP_CODE_DEFLATE,6)
status = sfwdata(sds_id, hstart, hstride, hedges, variable)
status = sfendacc(sds_id)
status = sfend(sd_id)

Both the codes worked fine. The only problem was that I got an hdf5
file which was at least 5 times the size of the hdf4 file. That
doesnt make sense. It should be the same right? No matter what value
I gave for chunkdim_sizes, the result was the same.

Can anyone please help

Thanks,
Abhilash Chandy.

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to

hdf-forum-subscribe@hdfgroup.org.

To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

--

------------------------------------------------------------
Elena Pourmal
The HDF Group
1901 So First ST.
Suite C-2
Champaign, IL 61820

epourmal@hdfgroup.org
(217)333-0238 (office)
(217)333-9049 (fax)
------------------------------------------------------------

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

     ____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

7

Please try chunk size 32x32x32

HDF5 applies compression to each chunk which is in this case 1 element (4 bytes), no compression is applied at all.

Elena

···

At 6:46 PM -0800 1/28/08, abhilash chandy wrote:

Thanks for the reply. Here is the output of h5dump

HDF5 "var.hdf" {
GROUP "/" {
   DATASET "var" {
      DATATYPE H5T_IEEE_F32LE
      DATASPACE SIMPLE { ( 32, 32, 32 ) / ( 32, 32, 32 ) }
      STORAGE_LAYOUT {
         CHUNKED ( 1, 1, 1 )
         SIZE 131072
       }
      FILTERS {
         COMPRESSION SZIP {
            PIXELS_PER_BLOCK 4
            MODE K13
            CODING NEAREST NEIGHBOUR
                        HEADER RAW
         }
      }
      FILLVALUE {
         FILL_TIME H5D_FILL_TIME_IFSET
         VALUE 0 }
      ALLOCATION_TIME {
         H5D_ALLOC_TIME_INCR
      }
   }
}

----- Original Message ----
From: Elena Pourmal <epourmal@hdfgroup.org>
To: abhilash chandy <abhilash_chandy@yahoo.com>; hdf-forum@hdfgroup.org
Sent: Monday, January 28, 2008 8:07:12 PM
Subject: Re: compression in hdf5

Abhilash,

If you data is small, size of the HDF5 overhead may be comparable
with the size of the compressed data.

Could you please send me the output of the

h5dump -p -H <yourfile.h5>
?

Thank you!

Elena

At 4:58 PM -0800 1/28/08, abhilash chandy wrote:

Hi,
   I am trying to test compression in hdf5 by testing a small program
which just writes out one variable after creating a dataset. It kind
of looks like this

call h5fcreate_f(filename, H5F_ACC_TRUNC_F, file_id, error)
call h5screate_simple_f(rank,dim_sizes,space_id,hstatus)
call h5pcreate_f(H5P_DATASET_CREATE_F, plist_id, hstatus)
call h5pset_chunk_f(plist_id,rank,chunkdim_sizes,hstatus)

!Set the compression parameters
comp_prm(1)=6 ! not the best, but close (and fast)
call h5pset_deflate_f(plist_id, comp_prm(1), hstatus)
call h5tcopy_f(H5T_NATIVE_REAL,type_id,hstatus)
call

h5dcreate_f(file_id,vname,type_id,space_id,sds_id,hstatus,plist_id)

call h5dwrite_f(sds_id,type_id,variable,dim_sizes,hstatus)

Ofourse this is in hdf5. I try the same thing in hdf4 and the sample
code looks like this

sd_id = sfstart(filename,DFACC_CREATE)
sds_id = sfcreate(sd_id,vname,DFNT_FLOAT32, rank, dim_sizes)
status = sfscompress(sds_id,COMP_CODE_DEFLATE,6)
status = sfwdata(sds_id, hstart, hstride, hedges, variable)
status = sfendacc(sds_id)
status = sfend(sd_id)

Both the codes worked fine. The only problem was that I got an hdf5
file which was at least 5 times the size of the hdf4 file. That
doesnt make sense. It should be the same right? No matter what value
I gave for chunkdim_sizes, the result was the same.

Can anyone please help

Thanks,
Abhilash Chandy.

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to

hdf-forum-subscribe@hdfgroup.org.

To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

--

------------------------------------------------------------
Elena Pourmal
The HDF Group
1901 So First ST.
Suite C-2
Champaign, IL 61820

epourmal@hdfgroup.org
(217)333-0238 (office)
(217)333-9049 (fax)
------------------------------------------------------------

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ

--

------------------------------------------------------------
Elena Pourmal
The HDF Group
1901 So First ST.
Suite C-2
Champaign, IL 61820

epourmal@hdfgroup.org
(217)333-0238 (office)
(217)333-9049 (fax)
------------------------------------------------------------

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.