Is it possible to change a datasets fillvalue, once established?

Hello,

I'm using v1.8.4-patched on Linux x86_64, and would like to know if
it is possible to change a fillvalue [ H5Pset_fill_value() ] for a
dataset, once it has already been "burned in" at creation time?

   Below I show code for trying to change an existing fillvalue using
HDF5 ANSI C API, and realize I'm missing something critical about how
H5Dget_create_plist( {YourDataset_ID} ) works.

   One motivation for this is that I'd like to use H5Py (v1.3.0) to
author the dataset, but H5Py does not yet support assigning user
defined (e.g. non "0") fillvalues as part of a dataset creation call.
    I have a large earth science dataset (in raw binary) that uses a
fillvalue of -9999.0. If I use H5Py v1.3.0 to initially create these
HDF5 files, they end up missing the fillvalue. as an example of the
problem, HDFView and Panoply "see" the pixels coded -9999.0 as valid
data, which seriously messes up the color palette rendering, among
other things... I've originally tried using various CF v1.4.x and
similar standards (e.g. _Fillvalue or missing_data) and verified these
are distinct from the internal "fillvalue" defined using
H5Pset_fill_value()...

I realize I could create the dataset using just the HDF5 C API, but
I've already got the H5Py code about 99% written ..except for this
lack of fill-value assignment...:))

My example code below shows how I've tried to re-assign a fillvalue
(on the assumption its possible):

   file_id = H5Fopen("test.h5",H5F_ACC_RDWR,H5P_DEFAULT) ;
   /* fetch the dataset id assoc with name "DS1" */
   dset = H5Dopen(file_id,"DS1",H5P_DEFAULT);

   /* hid_t H5Dget_create_plist (hid_t dataset_id)
       Returns an identifier for a copy of the dataset creation property list.
       The new property list is created by examining various permanent
properties
       of the dataset. This is mostly a catch-all for everything but
type and space. */

    int fillval = -8 ;
    dcpl = H5Dget_create_plist(dset);
    status = H5Pset_fill_value (dcpl, H5T_NATIVE_INT, &fillVal);

    /* this superfiscially shows the assignment (via dcpl), per se,
worked correctly...*/
    status = H5Pget_fill_value(dcpl,H5T_NATIVE_INT,&chkfill);
    printf("Status after H5Pget_fill_value %d chkfill %d \n",status,chkfill);

    /* but, after the HDF5 file is closed, the new fill value in dcpl
DOES NOT propogate
       to the physical HDF5 file as hoped...*/

  Perhaps someone could point out what's possible and/or where I've
gone wrong, and indicate a work-around....

thanks much in advance

jmg

···

------------------------------------------------------------
Joseph Glassy
Lead Software Engineer (contractor)
NASA Measures (Freeze/Thaw),Rm CFC 424
College of Forestry and Conservation
Univ. Montana, Missoula, MT 59812

Hi Joseph,

Hello,

I'm using v1.8.4-patched on Linux x86_64, and would like to know if
it is possible to change a fillvalue [ H5Pset_fill_value() ] for a
dataset, once it has already been "burned in" at creation time?

  No, dataset/group/file creation properties are permanent once the object exists.

  Below I show code for trying to change an existing fillvalue using
HDF5 ANSI C API, and realize I'm missing something critical about how
H5Dget_create_plist( {YourDataset_ID} ) works.

  The H5<X>get_create_plist call(s) return a _copy_ of the creation property list for an object. Changing the properties in that list has no effect on the object.

  If you email Francesc Alted (the PyTables author), he might patch in support for setting the fill value of a dataset.

  Quincey

···

On Sep 7, 2010, at 8:09 PM, J Glassy wrote:

  One motivation for this is that I'd like to use H5Py (v1.3.0) to
author the dataset, but H5Py does not yet support assigning user
defined (e.g. non "0") fillvalues as part of a dataset creation call.
   I have a large earth science dataset (in raw binary) that uses a
fillvalue of -9999.0. If I use H5Py v1.3.0 to initially create these
HDF5 files, they end up missing the fillvalue. as an example of the
problem, HDFView and Panoply "see" the pixels coded -9999.0 as valid
data, which seriously messes up the color palette rendering, among
other things... I've originally tried using various CF v1.4.x and
similar standards (e.g. _Fillvalue or missing_data) and verified these
are distinct from the internal "fillvalue" defined using
H5Pset_fill_value()...

I realize I could create the dataset using just the HDF5 C API, but
I've already got the H5Py code about 99% written ..except for this
lack of fill-value assignment...:))

My example code below shows how I've tried to re-assign a fillvalue
(on the assumption its possible):

  file_id = H5Fopen("test.h5",H5F_ACC_RDWR,H5P_DEFAULT) ;
  /* fetch the dataset id assoc with name "DS1" */
  dset = H5Dopen(file_id,"DS1",H5P_DEFAULT);

  /* hid_t H5Dget_create_plist (hid_t dataset_id)
      Returns an identifier for a copy of the dataset creation property list.
      The new property list is created by examining various permanent
properties
      of the dataset. This is mostly a catch-all for everything but
type and space. */

   int fillval = -8 ;
   dcpl = H5Dget_create_plist(dset);
   status = H5Pset_fill_value (dcpl, H5T_NATIVE_INT, &fillVal);

   /* this superfiscially shows the assignment (via dcpl), per se,
worked correctly...*/
   status = H5Pget_fill_value(dcpl,H5T_NATIVE_INT,&chkfill);
   printf("Status after H5Pget_fill_value %d chkfill %d \n",status,chkfill);

   /* but, after the HDF5 file is closed, the new fill value in dcpl
DOES NOT propogate
      to the physical HDF5 file as hoped...*/

Perhaps someone could point out what's possible and/or where I've
gone wrong, and indicate a work-around....

thanks much in advance

jmg

------------------------------------------------------------
Joseph Glassy
Lead Software Engineer (contractor)
NASA Measures (Freeze/Thaw),Rm CFC 424
College of Forestry and Conservation
Univ. Montana, Missoula, MT 59812

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

A Wednesday 08 September 2010 09:29:53 Quincey Koziol escrigué:

Hi Joseph,

> Hello,
>
> I'm using v1.8.4-patched on Linux x86_64, and would like to know if
> it is possible to change a fillvalue [ H5Pset_fill_value() ] for a
> dataset, once it has already been "burned in" at creation time?

  No, dataset/group/file creation properties are permanent once the object
exists.

> Below I show code for trying to change an existing fillvalue using
> HDF5 ANSI C API, and realize I'm missing something critical about how
> H5Dget_create_plist( {YourDataset_ID} ) works.

  The H5<X>get_create_plist call(s) return a _copy_ of the creation property
list for an object. Changing the properties in that list has no effect on
the object.

  If you email Francesc Alted (the PyTables author), he might patch in
support for setting the fill value of a dataset.

Err, I think you meant Andrew Collette (the author of h5py ;-). BTW, PyTables
already has support for setting these fill values in creation time.

···

On Sep 7, 2010, at 8:09 PM, J Glassy wrote:

--
Francesc Alted

Whoops! Sorry, I have to read through the email I'm replying to closer! :slight_smile:

    Quincey

···

On Sep 8, 2010, at 9:45 AM, Francesc Alted wrote:

A Wednesday 08 September 2010 09:29:53 Quincey Koziol escrigué:

Hi Joseph,

On Sep 7, 2010, at 8:09 PM, J Glassy wrote:

Hello,

I'm using v1.8.4-patched on Linux x86_64, and would like to know if
it is possible to change a fillvalue [ H5Pset_fill_value() ] for a
dataset, once it has already been "burned in" at creation time?

  No, dataset/group/file creation properties are permanent once the object
exists.

Below I show code for trying to change an existing fillvalue using
HDF5 ANSI C API, and realize I'm missing something critical about how
H5Dget_create_plist( {YourDataset_ID} ) works.

  The H5<X>get_create_plist call(s) return a _copy_ of the creation property
list for an object. Changing the properties in that list has no effect on
the object.

  If you email Francesc Alted (the PyTables author), he might patch in
support for setting the fill value of a dataset.

Err, I think you meant Andrew Collette (the author of h5py ;-). BTW, PyTables
already has support for setting these fill values in creation time.