scale-offset not portable between BE and LE machine

Hi,

I have run into a problem where scale-offset compressed data on a big endian
machine is uncompressed perfectly on the BE machine, but is messed up when
read on a LE machine (including output of stuff like h5dump).

Here is the (correct) output on the BE machine (64 bit IBM Power6) which
wrote the data:
bp-login2:/u/home/ac/orf/work15/run/so-only.00480.cdir% uname -a
AIX bp-login2 1 6 00C06B924C00

bp-login2:/u/home/ac/orf/work15/run/so-only.00480.cdir% h5dump -d
/3d/vinterp so-only.00480_0009.cm1hdf5|head
HDF5 "so-only.00480_0009.cm1hdf5" {
DATASET "/3d/vinterp" {
   DATATYPE H5T_IEEE_F32BE
   DATASPACE SIMPLE { ( 40, 30, 30 ) / ( 40, 30, 30 ) }
   DATA {
   (0,0,0): -0.0837158, -0.0837158, -0.0837158, -0.0837158, -0.0827158,
   (0,0,5): -0.0827158, -0.0827158, -0.0817158, -0.0807158, -0.0787158,
   (0,0,10): -0.0767158, -0.0727158, -0.0627158, -0.0407158, -0.00571576,
   (0,0,15): 0.0372842, 0.0732842, 0.0942842, 0.0982842, 0.0902842,
   (0,0,20): 0.0722842, 0.0492842, 0.0282842, 0.0122842, 0.00928424,
   (0,0,25): 0.0242842, 0.0562842, 0.0892842, 0.112284, 0.129284,
   (0,1,0): -0.0847158, -0.0847158, -0.0847158, -0.0847158, -0.0847158,

The same file, moved to a LE machine (32 bit intel):
seagrape:/users/orf/work15/compare% uname -a
Linux seagrape 2.6.18-194.26.1.el5PAE #1 SMP Tue Nov 9 13:34:42 EST 2010
i686 i686 i386 GNU/Linux

seagrape:/users/orf/work15/compare% h5dump -d /3d/vinterp
so-only.00480_0009.cm1hdf5|head
HDF5 "so-only.00480_0009.cm1hdf5" {
DATASET "/3d/vinterp" {
   DATATYPE H5T_IEEE_F32BE
   DATASPACE SIMPLE { ( 40, 30, 30 ) / ( 40, 30, 30 ) }
   DATA {
   (0,0,0): 1.012, 1.012, 1.012, 1.012, 1.013, 1.013, 1.013, 1.014, 1.015,
   (0,0,9): 1.017, 1.019, 1.023, 1.033, 1.055, 1.09, 1.133, 1.169, 1.19,
   (0,0,18): 1.194, 1.186, 1.168, 1.145, 1.124, 1.108, 1.105, 1.12, 1.152,
   (0,0,27): 1.185, 1.208, 1.225,
   (0,1,0): 1.011, 1.011, 1.011, 1.011, 1.011, 1.011, 1.011, 1.012, 1.012,

It is worth noting that no negative values are ever read on the LE machine
for any variable which really is negative.

The BE machine has Version 1.8.4-patch1 while the LE machine has Version
1.8.5-patch1. I am currently unable to build 1.8.5-patch1 on the BE machine
(internal compiler error, fun fun, I love AIX).

Data was written as such:

      call h5screate_simple_f(rank,dims,dspace_id,ierror)
      call h5pcreate_f(H5P_DATASET_CREATE_F,chunk_id,ierror)
      call h5pset_chunk_f(chunk_id,rank,chunkdims,ierror)
      call h5pset_scaleoffset_f (chunk_id,0,scale_factor,ierror)
      call
h5dcreate_f(file_id,trim(varname),H5T_NATIVE_REAL,dspace_id,dset_id,ierror,chunk_id)
      call
h5dwrite_f(dset_id,H5T_NATIVE_REAL,fl_array_3d,dims,ierror,H5S_ALL_F,H5S_ALL_F,H5P_DEFAULT_F)

I do not have these problems when using gzip compression or no compression.

Leigh

···

--
Leigh Orf
Associate Professor of Atmospheric Science
Department of Geology and Meteorology
Central Michigan University
Currently on sabbatical at the National Center for Atmospheric Research
in Boulder, CO
NCAR office phone: (303) 497-8200

Hi Leigh,

Hmm... doesn't sound right....

Could you please send me your file or upload it to ftp://ftp.hdfgroup.uiuc.edu/pub/incoming/epourmal directory?

We do test on bp at NCSA. What kind of errors are getting when you try to build?

Thank you!

Elena

···

On Jan 13, 2011, at 2:39 PM, Leigh Orf wrote:

Hi,

I have run into a problem where scale-offset compressed data on a big endian machine is uncompressed perfectly on the BE machine, but is messed up when read on a LE machine (including output of stuff like h5dump).

Here is the (correct) output on the BE machine (64 bit IBM Power6) which wrote the data:
bp-login2:/u/home/ac/orf/work15/run/so-only.00480.cdir% uname -a
AIX bp-login2 1 6 00C06B924C00

bp-login2:/u/home/ac/orf/work15/run/so-only.00480.cdir% h5dump -d /3d/vinterp so-only.00480_0009.cm1hdf5|head
HDF5 "so-only.00480_0009.cm1hdf5" {
DATASET "/3d/vinterp" {
   DATATYPE H5T_IEEE_F32BE
   DATASPACE SIMPLE { ( 40, 30, 30 ) / ( 40, 30, 30 ) }
   DATA {
   (0,0,0): -0.0837158, -0.0837158, -0.0837158, -0.0837158, -0.0827158,
   (0,0,5): -0.0827158, -0.0827158, -0.0817158, -0.0807158, -0.0787158,
   (0,0,10): -0.0767158, -0.0727158, -0.0627158, -0.0407158, -0.00571576,
   (0,0,15): 0.0372842, 0.0732842, 0.0942842, 0.0982842, 0.0902842,
   (0,0,20): 0.0722842, 0.0492842, 0.0282842, 0.0122842, 0.00928424,
   (0,0,25): 0.0242842, 0.0562842, 0.0892842, 0.112284, 0.129284,
   (0,1,0): -0.0847158, -0.0847158, -0.0847158, -0.0847158, -0.0847158,

The same file, moved to a LE machine (32 bit intel):
seagrape:/users/orf/work15/compare% uname -a
Linux seagrape 2.6.18-194.26.1.el5PAE #1 SMP Tue Nov 9 13:34:42 EST 2010 i686 i686 i386 GNU/Linux

seagrape:/users/orf/work15/compare% h5dump -d /3d/vinterp so-only.00480_0009.cm1hdf5|head
HDF5 "so-only.00480_0009.cm1hdf5" {
DATASET "/3d/vinterp" {
   DATATYPE H5T_IEEE_F32BE
   DATASPACE SIMPLE { ( 40, 30, 30 ) / ( 40, 30, 30 ) }
   DATA {
   (0,0,0): 1.012, 1.012, 1.012, 1.012, 1.013, 1.013, 1.013, 1.014, 1.015,
   (0,0,9): 1.017, 1.019, 1.023, 1.033, 1.055, 1.09, 1.133, 1.169, 1.19,
   (0,0,18): 1.194, 1.186, 1.168, 1.145, 1.124, 1.108, 1.105, 1.12, 1.152,
   (0,0,27): 1.185, 1.208, 1.225,
   (0,1,0): 1.011, 1.011, 1.011, 1.011, 1.011, 1.011, 1.011, 1.012, 1.012,

It is worth noting that no negative values are ever read on the LE machine for any variable which really is negative.

The BE machine has Version 1.8.4-patch1 while the LE machine has Version 1.8.5-patch1. I am currently unable to build 1.8.5-patch1 on the BE machine (internal compiler error, fun fun, I love AIX).

Data was written as such:

      call h5screate_simple_f(rank,dims,dspace_id,ierror)
      call h5pcreate_f(H5P_DATASET_CREATE_F,chunk_id,ierror)
      call h5pset_chunk_f(chunk_id,rank,chunkdims,ierror)
      call h5pset_scaleoffset_f (chunk_id,0,scale_factor,ierror)
      call h5dcreate_f(file_id,trim(varname),H5T_NATIVE_REAL,dspace_id,dset_id,ierror,chunk_id)
      call h5dwrite_f(dset_id,H5T_NATIVE_REAL,fl_array_3d,dims,ierror,H5S_ALL_F,H5S_ALL_F,H5P_DEFAULT_F)

I do not have these problems when using gzip compression or no compression.

Leigh

--
Leigh Orf
Associate Professor of Atmospheric Science
Department of Geology and Meteorology
Central Michigan University
Currently on sabbatical at the National Center for Atmospheric Research in Boulder, CO
NCAR office phone: (303) 497-8200

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Leigh,

Thank you for reporting a bug. We just confirmed that it is in both 1.8.5-patch 1 and 1.8.6.

We will try to address it ASAP, but I am afraid, we will not be able to get it into the 1.8.6 release.

Elena

···

On Jan 13, 2011, at 2:39 PM, Leigh Orf wrote:

Hi,

I have run into a problem where scale-offset compressed data on a big endian machine is uncompressed perfectly on the BE machine, but is messed up when read on a LE machine (including output of stuff like h5dump).

Here is the (correct) output on the BE machine (64 bit IBM Power6) which wrote the data:
bp-login2:/u/home/ac/orf/work15/run/so-only.00480.cdir% uname -a
AIX bp-login2 1 6 00C06B924C00

bp-login2:/u/home/ac/orf/work15/run/so-only.00480.cdir% h5dump -d /3d/vinterp so-only.00480_0009.cm1hdf5|head
HDF5 "so-only.00480_0009.cm1hdf5" {
DATASET "/3d/vinterp" {
   DATATYPE H5T_IEEE_F32BE
   DATASPACE SIMPLE { ( 40, 30, 30 ) / ( 40, 30, 30 ) }
   DATA {
   (0,0,0): -0.0837158, -0.0837158, -0.0837158, -0.0837158, -0.0827158,
   (0,0,5): -0.0827158, -0.0827158, -0.0817158, -0.0807158, -0.0787158,
   (0,0,10): -0.0767158, -0.0727158, -0.0627158, -0.0407158, -0.00571576,
   (0,0,15): 0.0372842, 0.0732842, 0.0942842, 0.0982842, 0.0902842,
   (0,0,20): 0.0722842, 0.0492842, 0.0282842, 0.0122842, 0.00928424,
   (0,0,25): 0.0242842, 0.0562842, 0.0892842, 0.112284, 0.129284,
   (0,1,0): -0.0847158, -0.0847158, -0.0847158, -0.0847158, -0.0847158,

The same file, moved to a LE machine (32 bit intel):
seagrape:/users/orf/work15/compare% uname -a
Linux seagrape 2.6.18-194.26.1.el5PAE #1 SMP Tue Nov 9 13:34:42 EST 2010 i686 i686 i386 GNU/Linux

seagrape:/users/orf/work15/compare% h5dump -d /3d/vinterp so-only.00480_0009.cm1hdf5|head
HDF5 "so-only.00480_0009.cm1hdf5" {
DATASET "/3d/vinterp" {
   DATATYPE H5T_IEEE_F32BE
   DATASPACE SIMPLE { ( 40, 30, 30 ) / ( 40, 30, 30 ) }
   DATA {
   (0,0,0): 1.012, 1.012, 1.012, 1.012, 1.013, 1.013, 1.013, 1.014, 1.015,
   (0,0,9): 1.017, 1.019, 1.023, 1.033, 1.055, 1.09, 1.133, 1.169, 1.19,
   (0,0,18): 1.194, 1.186, 1.168, 1.145, 1.124, 1.108, 1.105, 1.12, 1.152,
   (0,0,27): 1.185, 1.208, 1.225,
   (0,1,0): 1.011, 1.011, 1.011, 1.011, 1.011, 1.011, 1.011, 1.012, 1.012,

It is worth noting that no negative values are ever read on the LE machine for any variable which really is negative.

The BE machine has Version 1.8.4-patch1 while the LE machine has Version 1.8.5-patch1. I am currently unable to build 1.8.5-patch1 on the BE machine (internal compiler error, fun fun, I love AIX).

Data was written as such:

      call h5screate_simple_f(rank,dims,dspace_id,ierror)
      call h5pcreate_f(H5P_DATASET_CREATE_F,chunk_id,ierror)
      call h5pset_chunk_f(chunk_id,rank,chunkdims,ierror)
      call h5pset_scaleoffset_f (chunk_id,0,scale_factor,ierror)
      call h5dcreate_f(file_id,trim(varname),H5T_NATIVE_REAL,dspace_id,dset_id,ierror,chunk_id)
      call h5dwrite_f(dset_id,H5T_NATIVE_REAL,fl_array_3d,dims,ierror,H5S_ALL_F,H5S_ALL_F,H5P_DEFAULT_F)

I do not have these problems when using gzip compression or no compression.

Leigh

--
Leigh Orf
Associate Professor of Atmospheric Science
Department of Geology and Meteorology
Central Michigan University
Currently on sabbatical at the National Center for Atmospheric Research in Boulder, CO
NCAR office phone: (303) 497-8200

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org