Errors when enabling compression for Variable length datatypes.

Dear HDF users,

I would like to use a variable length datatype, which I would like to store
in a dataset with compression enabled (I am using HDF 1.8.2)

Unfortunately, I am unable to create the dataset when dataset create
properties set to compression. I am getting the following error message.
.HDF5-DIAG: Error detected in HDF5 (1.8.2) thread 0:
  #000: ..\..\..\src\H5D.c line 171 in H5Dcreate2(): unable to create
dataset
    major: Dataset
    minor: Unable to initialize object
... etc (for entire error message see bottom of e-mail).

The same code with Data-set create properties set to H5P_DEFAULT works fine.
I also can't create a dataset when I use vl datatypes and set maxdims in
H5Screate_simpel to H5S_UNLIMITED

The code causing the error messages looks like that:

hid_t peaklist = H5Tvlen_create(H5T_NATIVE_DOUBLE);

hid_t h5file_ = H5Fcreate("testfile.h5" , H5F_ACC_TRUNC, H5P_DEFAULT,
H5P_DEFAULT);

                std::string datasetname("Array");
                const int length = 10000;
                const int RANK = 1;
                hsize_t dim[] = {length}; /* Dataspace dimensions */

                hsize_t maxdims[] = {H5S_UNLIMITED};
                hid_t dataspace_ = H5Screate_simple(RANK, dim, NULL);
//hid_t datasepace_ = H5Screate_simple(RANK, dim, maxdims); //creates error.
                /*!
                * Create the dataset.
                */
                hid_t dscreateProperties_ = H5Pcreate(H5P_DATASET_CREATE);
                H5Pset_szip(dscreateProperties_
                    , H5_SZIP_NN_OPTION_MASK , 16 );
                hid_t dataset_ = H5Dcreate2
                    (
                    h5file_,
                    datasetname.c_str(),
                    peaklist,
                    dataspace_,
                    H5P_DEFAULT,
                    dscreateProperties_, //does not work
// H5P_DEFAULT, // works
                    H5P_DEFAULT
                    );

                exit(0);

Is compression incompatible with variable datasets?

Thank you
Eryk

Error message:
.HDF5-DIAG: Error detected in HDF5 (1.8.2) thread 0:
  #000: ..\..\..\src\H5D.c line 171 in H5Dcreate2(): unable to create
dataset
    major: Dataset
    minor: Unable to initialize object
  #001: ..\..\..\src\H5Dint.c line 429 in H5D_create_named(): unable to
create and link to dataset
    major: Dataset
    minor: Unable to initialize object
  #002: ..\..\..\src\H5L.c line 1633 in H5L_link_object(): unable to create
new link to object
    major: Links
    minor: Unable to initialize object
  #003: ..\..\..\src\H5L.c line 1856 in H5L_create_real(): can't insert link
    major: Symbol table
    minor: Unable to insert object
  #004: ..\..\..\src\H5Gtraverse.c line 877 in H5G_traverse(): internal path
traversal failed
    major: Symbol table
    minor: Object not found
  #005: ..\..\..\src\H5Gtraverse.c line 703 in H5G_traverse_real():
traversal operator failed
    major: Symbol table
    minor: Callback failed
  #006: ..\..\..\src\H5L.c line 1679 in H5L_link_cb(): unable to create
object
    major: Object header
    minor: Unable to initialize object
  #007: ..\..\..\src\H5O.c line 2531 in H5O_obj_create(): unable to open
object
    major: Object header
    minor: Can't open object
  #008: ..\..\..\src\H5Doh.c line 278 in H5O_dset_create(): unable to create
dataset
    major: Dataset
    minor: Unable to initialize object
  #009: ..\..\..\src\H5Dint.c line 1089 in H5D_create(): filters can only be
used with chunked layout
    major: Dataset
    minor: Bad value

···

--
Witold Eryk Wolski

Heidmark str 5
D-28329 Bremen
tel.: 04215261837

I see several problems here:

1) You are not storing a variable length type. H5T_NATIVE_DOUBLE will
always be a fixed length type. If you want to be able to change the
size of an array/dataset after storing it, this done through resizing
the dataset. Variable length refers to items of different lengths
being stored in a single dataset, not to varying the size of a dataset.
The most common variable length item would be strings.

2) The error message tells you exactly why compression cannot be
applied, it tells you that it only works when you are chunking data. If
you don't understand this, you probably need to spend more time on the
documentation, like here:

http://www.hdfgroup.org/HDF5/Tutor/property.html#cd

3) This is not an error, but an observation: When you store variable
length data in HDF5, what is actually stored in the dataset itself is a
pointer to the data. When you apply compression, only the pointer data
would be compressed, not the data itself.

George Lewandowski
(314)777-7890
Mail Code S270-2204
Building 270-E Level 2E Room 20E
P-8A

···

________________________________

From: W Eryk Wolski [mailto:wewolski@gmail.com]
Sent: Monday, July 27, 2009 11:52 AM
To: HDF forum
Subject: [Hdf-forum] Errors when enabling compression for Variable
lengthdatatypes.

Dear HDF users,

I would like to use a variable length datatype, which I would like to
store in a dataset with compression enabled (I am using HDF 1.8.2)

Unfortunately, I am unable to create the dataset when dataset create
properties set to compression. I am getting the following error message.
.HDF5-DIAG: Error detected in HDF5 (1.8.2) thread 0:
  #000: ..\..\..\src\H5D.c line 171 in H5Dcreate2(): unable to create
dataset
    major: Dataset
    minor: Unable to initialize object
... etc (for entire error message see bottom of e-mail).

The same code with Data-set create properties set to H5P_DEFAULT works
fine.
I also can't create a dataset when I use vl datatypes and set maxdims in
H5Screate_simpel to H5S_UNLIMITED

The code causing the error messages looks like that:

hid_t peaklist = H5Tvlen_create(H5T_NATIVE_DOUBLE);

hid_t h5file_ = H5Fcreate("testfile.h5" , H5F_ACC_TRUNC, H5P_DEFAULT,
H5P_DEFAULT);

                std::string datasetname("Array");
                const int length = 10000;
                const int RANK = 1;
                hsize_t dim[] = {length}; /* Dataspace dimensions
*/

                hsize_t maxdims[] = {H5S_UNLIMITED};
                hid_t dataspace_ = H5Screate_simple(RANK, dim, NULL);
//hid_t datasepace_ = H5Screate_simple(RANK, dim, maxdims); //creates
error.
                /*!
                * Create the dataset.
                */
                hid_t dscreateProperties_ =
H5Pcreate(H5P_DATASET_CREATE);
                H5Pset_szip(dscreateProperties_
                    , H5_SZIP_NN_OPTION_MASK , 16 );
                hid_t dataset_ = H5Dcreate2
                    (
                    h5file_,
                    datasetname.c_str(),
                    peaklist,
                    dataspace_,
                    H5P_DEFAULT,
                    dscreateProperties_, //does not work
// H5P_DEFAULT, // works
                    H5P_DEFAULT
                    );

                exit(0);

Is compression incompatible with variable datasets?

Thank you
Eryk

Error message:
.HDF5-DIAG: Error detected in HDF5 (1.8.2) thread 0:
  #000: ..\..\..\src\H5D.c line 171 in H5Dcreate2(): unable to create
dataset
    major: Dataset
    minor: Unable to initialize object
  #001: ..\..\..\src\H5Dint.c line 429 in H5D_create_named(): unable to
create and link to dataset
    major: Dataset
    minor: Unable to initialize object
  #002: ..\..\..\src\H5L.c line 1633 in H5L_link_object(): unable to
create new link to object
    major: Links
    minor: Unable to initialize object
  #003: ..\..\..\src\H5L.c line 1856 in H5L_create_real(): can't insert
link
    major: Symbol table
    minor: Unable to insert object
  #004: ..\..\..\src\H5Gtraverse.c line 877 in H5G_traverse(): internal
path traversal failed
    major: Symbol table
    minor: Object not found
  #005: ..\..\..\src\H5Gtraverse.c line 703 in H5G_traverse_real():
traversal operator failed
    major: Symbol table
    minor: Callback failed
  #006: ..\..\..\src\H5L.c line 1679 in H5L_link_cb(): unable to create
object
    major: Object header
    minor: Unable to initialize object
  #007: ..\..\..\src\H5O.c line 2531 in H5O_obj_create(): unable to open
object
    major: Object header
    minor: Can't open object
  #008: ..\..\..\src\H5Doh.c line 278 in H5O_dset_create(): unable to
create dataset
    major: Dataset
    minor: Unable to initialize object
  #009: ..\..\..\src\H5Dint.c line 1089 in H5D_create(): filters can
only be used with chunked layout
    major: Dataset
    minor: Bad value

--
Witold Eryk Wolski

Heidmark str 5
D-28329 Bremen
tel.: 04215261837

George.

I do really appreciate the responses on this list. Thanks to all of you for
them. I do know that it takes time and attention to understand the problems
posted and to provide answers. Therefore, I did spend time and effort to
read the doc, test the code, and afterward make it clear and easy to
understand what the problem is I have. I am sorry if the exposition of the
problem I provided was'nt helpfull.

As far as I understood you points George (and to clarify the question asked
)

Ad 1) I do store a variable length type (peaklist) and not as you have
understood a fixed length type H5T_NATIVE_DOUBLE. The peaklist type is
defined as follows:
hid_t peaklist = H5Tvlen_create(H5T_NATIVE_DOUBLE); //defines a vlen
datatype

hid_t dataset_ = H5Dcreate2
                    (
                    h5file_,
                    datasetname.c_str(),
                    peaklist, // <<< and peaklist is the datatype I
store....

Ad 2) Your are right, chunking is required for compression. The original
code did enable chunking. I just was aiming to make the posted code as short
as possible still reproducing the error. The original code enabling chunking
looks like this:

hid_t dscreateProperties_ = H5Pcreate(H5P_DATASET_CREATE);
                hsize_t chunk_dims_[] = {1024};
                H5Pset_chunk(dscreateProperties_, RANK, chunk_dims_ );
                H5Pset_szip(dscreateProperties_
                    , H5_SZIP_NN_OPTION_MASK , 16 );

And produces exactly the same error messages as posted yesterday with vlen
datatypes. Still the dataset can't be created and the error messages are the
same. So it is not down to chunking but down to vlen that the dataset can't
be created.

Ad 3) This is you guess isn't it? It would be my guess too.

This might explain while compression can't be applied to vlen datatypes.
Hovewer, I did observed the following problem I did not posted yesterday.
Assuming compression disabled but trying to enable extensible datasets...

hsize_t maxdims[] = {H5S_UNLIMITED};
hid_t dataspace_ = H5Screate_simple(RANK, dim, NULL); //with this setting
dataset can be created
hid_t dataspace_ = H5Screate_simple(RANK, dim, maxdims); // creation of
dataset FAILS.

Is it possible to enable compression with vlen's? Why vlen datasets can't be
extensible???

regards
Eryk

···

2009/7/27 Lewandowski, George <george.lewandowski@boeing.com>

I see several problems here:

1) You are not storing a variable length type. H5T_NATIVE_DOUBLE will
always be a fixed length type. If you want to be able to change the size of
an array/dataset after storing it, this done through resizing the dataset.
Variable length refers to items of different lengths being stored in a
single dataset, not to varying the size of a dataset. The most common
variable length item would be strings.

2) The error message tells you exactly why compression cannot be applied,
it tells you that it only works when you are chunking data. If you don't
understand this, you probably need to spend more time on the documentation,
like here:

http://www.hdfgroup.org/HDF5/Tutor/property.html#cd

3) This is not an error, but an observation: When you store variable
length data in HDF5, what is actually stored in the dataset itself is a
pointer to the data. When you apply compression, only the pointer data
would be compressed, not the data itself.

George Lewandowski
(314)777-7890
Mail Code S270-2204
Building 270-E Level 2E Room 20E
P-8A

------------------------------
*From:* W Eryk Wolski [mailto:wewolski@gmail.com]
*Sent:* Monday, July 27, 2009 11:52 AM
*To:* HDF forum
*Subject:* [Hdf-forum] Errors when enabling compression for Variable
lengthdatatypes.

Dear HDF users,

I would like to use a variable length datatype, which I would like to store
in a dataset with compression enabled (I am using HDF 1.8.2)

Unfortunately, I am unable to create the dataset when dataset create
properties set to compression. I am getting the following error message.
.HDF5-DIAG: Error detected in HDF5 (1.8.2) thread 0:
  #000: ..\..\..\src\H5D.c line 171 in H5Dcreate2(): unable to create
dataset
    major: Dataset
    minor: Unable to initialize object
... etc (for entire error message see bottom of e-mail).

The same code with Data-set create properties set to H5P_DEFAULT works
fine.
I also can't create a dataset when I use vl datatypes and set maxdims in
H5Screate_simpel to H5S_UNLIMITED

The code causing the error messages looks like that:

hid_t peaklist = H5Tvlen_create(H5T_NATIVE_DOUBLE);

hid_t h5file_ = H5Fcreate("testfile.h5" , H5F_ACC_TRUNC, H5P_DEFAULT,
H5P_DEFAULT);

                std::string datasetname("Array");
                const int length = 10000;
                const int RANK = 1;
                hsize_t dim[] = {length}; /* Dataspace dimensions */

                hsize_t maxdims[] = {H5S_UNLIMITED};
                hid_t dataspace_ = H5Screate_simple(RANK, dim, NULL);
//hid_t datasepace_ = H5Screate_simple(RANK, dim, maxdims); //creates
error.
                /*!
                * Create the dataset.
                */
                hid_t dscreateProperties_ = H5Pcreate(H5P_DATASET_CREATE);
                H5Pset_szip(dscreateProperties_
                    , H5_SZIP_NN_OPTION_MASK , 16 );
                hid_t dataset_ = H5Dcreate2
                    (
                    h5file_,
                    datasetname.c_str(),
                    peaklist,
                    dataspace_,
                    H5P_DEFAULT,
                    dscreateProperties_, //does not work
// H5P_DEFAULT, // works
                    H5P_DEFAULT
                    );

                exit(0);

Is compression incompatible with variable datasets?

Thank you
Eryk

Error message:
.HDF5-DIAG: Error detected in HDF5 (1.8.2) thread 0:
  #000: ..\..\..\src\H5D.c line 171 in H5Dcreate2(): unable to create
dataset
    major: Dataset
    minor: Unable to initialize object
  #001: ..\..\..\src\H5Dint.c line 429 in H5D_create_named(): unable to
create and link to dataset
    major: Dataset
    minor: Unable to initialize object
  #002: ..\..\..\src\H5L.c line 1633 in H5L_link_object(): unable to create
new link to object
    major: Links
    minor: Unable to initialize object
  #003: ..\..\..\src\H5L.c line 1856 in H5L_create_real(): can't insert
link
    major: Symbol table
    minor: Unable to insert object
  #004: ..\..\..\src\H5Gtraverse.c line 877 in H5G_traverse(): internal
path traversal failed
    major: Symbol table
    minor: Object not found
  #005: ..\..\..\src\H5Gtraverse.c line 703 in H5G_traverse_real():
traversal operator failed
    major: Symbol table
    minor: Callback failed
  #006: ..\..\..\src\H5L.c line 1679 in H5L_link_cb(): unable to create
object
    major: Object header
    minor: Unable to initialize object
  #007: ..\..\..\src\H5O.c line 2531 in H5O_obj_create(): unable to open
object
    major: Object header
    minor: Can't open object
  #008: ..\..\..\src\H5Doh.c line 278 in H5O_dset_create(): unable to
create dataset
    major: Dataset
    minor: Unable to initialize object
  #009: ..\..\..\src\H5Dint.c line 1089 in H5D_create(): filters can only
be used with chunked layout
    major: Dataset
    minor: Bad value

--
Witold Eryk Wolski

Heidmark str 5
D-28329 Bremen
tel.: 04215261837

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Witold Eryk Wolski

Heidmark str 5
D-28329 Bremen
tel.: 04215261837

Okay, I guess I didn't read the code example carefully enough. Someone
smarter than I am will have to respond about chunking and vlen, I don't
know how that is done.

On my third point, I know for sure that for variable length data, the
dataset itself contains a fixed-length structure that points to the
actual vlen data which is stored elsewhere. I also know for sure that
compression only applies to the pointer structures in the dataset
itself.

George Lewandowski
(314)777-7890
Mail Code S270-2204
Building 270-E Level 2E Room 20E
P-8A

···

________________________________

From: W Eryk Wolski [mailto:wewolski@gmail.com]
Sent: Tuesday, July 28, 2009 5:29 AM
To: hdf-forum@hdfgroup.org
Subject: Re: [Hdf-forum] Errors when enabling compression for
Variablelengthdatatypes.

George.

I do really appreciate the responses on this list. Thanks to all of you
for them. I do know that it takes time and attention to understand the
problems posted and to provide answers. Therefore, I did spend time and
effort to read the doc, test the code, and afterward make it clear and
easy to understand what the problem is I have. I am sorry if the
exposition of the problem I provided was'nt helpfull.

As far as I understood you points George (and to clarify the question
asked )

Ad 1) I do store a variable length type (peaklist) and not as you have
understood a fixed length type H5T_NATIVE_DOUBLE. The peaklist type is
defined as follows:
hid_t peaklist = H5Tvlen_create(H5T_NATIVE_DOUBLE); //defines a vlen
datatype

hid_t dataset_ = H5Dcreate2
                    (
                    h5file_,
                    datasetname.c_str(),
                    peaklist, // <<< and peaklist is the datatype I
store....

Ad 2) Your are right, chunking is required for compression. The original
code did enable chunking. I just was aiming to make the posted code as
short as possible still reproducing the error. The original code
enabling chunking looks like this:

hid_t dscreateProperties_ = H5Pcreate(H5P_DATASET_CREATE);
                hsize_t chunk_dims_[] = {1024};
                H5Pset_chunk(dscreateProperties_, RANK, chunk_dims_ );
                H5Pset_szip(dscreateProperties_
                    , H5_SZIP_NN_OPTION_MASK , 16 );

And produces exactly the same error messages as posted yesterday with
vlen datatypes. Still the dataset can't be created and the error
messages are the same. So it is not down to chunking but down to vlen
that the dataset can't be created.

Ad 3) This is you guess isn't it? It would be my guess too.

This might explain while compression can't be applied to vlen datatypes.
Hovewer, I did observed the following problem I did not posted
yesterday. Assuming compression disabled but trying to enable
extensible datasets...

hsize_t maxdims[] = {H5S_UNLIMITED};
hid_t dataspace_ = H5Screate_simple(RANK, dim, NULL); //with this
setting dataset can be created
hid_t dataspace_ = H5Screate_simple(RANK, dim, maxdims); // creation of
dataset FAILS.

Is it possible to enable compression with vlen's? Why vlen datasets
can't be extensible???

regards
Eryk

2009/7/27 Lewandowski, George <george.lewandowski@boeing.com>

  I see several problems here:
   
  1) You are not storing a variable length type.
H5T_NATIVE_DOUBLE will always be a fixed length type. If you want to be
able to change the size of an array/dataset after storing it, this done
through resizing the dataset. Variable length refers to items of
different lengths being stored in a single dataset, not to varying the
size of a dataset. The most common variable length item would be
strings.

  2) The error message tells you exactly why compression cannot be
applied, it tells you that it only works when you are chunking data. If
you don't understand this, you probably need to spend more time on the
documentation, like here:
   
  http://www.hdfgroup.org/HDF5/Tutor/property.html#cd
   
  3) This is not an error, but an observation: When you store
variable length data in HDF5, what is actually stored in the dataset
itself is a pointer to the data. When you apply compression, only the
pointer data would be compressed, not the data itself.
   
  George Lewandowski
  (314)777-7890
  Mail Code S270-2204
  Building 270-E Level 2E Room 20E
  P-8A

________________________________

  From: W Eryk Wolski [mailto:wewolski@gmail.com]
  Sent: Monday, July 27, 2009 11:52 AM
  To: HDF forum
  Subject: [Hdf-forum] Errors when enabling compression for
Variable lengthdatatypes.
  
  Dear HDF users,
  
  I would like to use a variable length datatype, which I would
like to store in a dataset with compression enabled (I am using HDF
1.8.2)
  
  Unfortunately, I am unable to create the dataset when dataset
create properties set to compression. I am getting the following error
message.
  .HDF5-DIAG: Error detected in HDF5 (1.8.2) thread 0:
    #000: ..\..\..\src\H5D.c line 171 in H5Dcreate2(): unable to
create dataset
      major: Dataset
      minor: Unable to initialize object
  ... etc (for entire error message see bottom of e-mail).
  
  The same code with Data-set create properties set to H5P_DEFAULT
works fine.
  I also can't create a dataset when I use vl datatypes and set
maxdims in H5Screate_simpel to H5S_UNLIMITED
  
  The code causing the error messages looks like that:
  
  hid_t peaklist = H5Tvlen_create(H5T_NATIVE_DOUBLE);
  
  hid_t h5file_ = H5Fcreate("testfile.h5" , H5F_ACC_TRUNC,
H5P_DEFAULT, H5P_DEFAULT);
  
                  std::string datasetname("Array");
                  const int length = 10000;
                  const int RANK = 1;
                  hsize_t dim[] = {length}; /* Dataspace
dimensions */
  
                  hsize_t maxdims[] = {H5S_UNLIMITED};
                  hid_t dataspace_ = H5Screate_simple(RANK, dim,
NULL);
  //hid_t datasepace_ = H5Screate_simple(RANK, dim, maxdims);
//creates error.
                  /*!
                  * Create the dataset.
                  */
                  hid_t dscreateProperties_ =
H5Pcreate(H5P_DATASET_CREATE);
                  H5Pset_szip(dscreateProperties_
                      , H5_SZIP_NN_OPTION_MASK , 16 );
                  hid_t dataset_ = H5Dcreate2
                      (
                      h5file_,
                      datasetname.c_str(),
                      peaklist,
                      dataspace_,
                      H5P_DEFAULT,
                      dscreateProperties_, //does not work
  // H5P_DEFAULT, // works
                      H5P_DEFAULT
                      );
  
                  exit(0);
  
  Is compression incompatible with variable datasets?
  
  Thank you
  Eryk
  
  Error message:
  .HDF5-DIAG: Error detected in HDF5 (1.8.2) thread 0:
    #000: ..\..\..\src\H5D.c line 171 in H5Dcreate2(): unable to
create dataset
      major: Dataset
      minor: Unable to initialize object
    #001: ..\..\..\src\H5Dint.c line 429 in H5D_create_named():
unable to create and link to dataset
      major: Dataset
      minor: Unable to initialize object
    #002: ..\..\..\src\H5L.c line 1633 in H5L_link_object():
unable to create new link to object
      major: Links
      minor: Unable to initialize object
    #003: ..\..\..\src\H5L.c line 1856 in H5L_create_real(): can't
insert link
      major: Symbol table
      minor: Unable to insert object
    #004: ..\..\..\src\H5Gtraverse.c line 877 in H5G_traverse():
internal path traversal failed
      major: Symbol table
      minor: Object not found
    #005: ..\..\..\src\H5Gtraverse.c line 703 in
H5G_traverse_real(): traversal operator failed
      major: Symbol table
      minor: Callback failed
    #006: ..\..\..\src\H5L.c line 1679 in H5L_link_cb(): unable to
create object
      major: Object header
      minor: Unable to initialize object
    #007: ..\..\..\src\H5O.c line 2531 in H5O_obj_create(): unable
to open object
      major: Object header
      minor: Can't open object
    #008: ..\..\..\src\H5Doh.c line 278 in H5O_dset_create():
unable to create dataset
      major: Dataset
      minor: Unable to initialize object
    #009: ..\..\..\src\H5Dint.c line 1089 in H5D_create(): filters
can only be used with chunked layout
      major: Dataset
      minor: Bad value
   
  --
  Witold Eryk Wolski
  
  Heidmark str 5
  D-28329 Bremen
  tel.: 04215261837
  
  _______________________________________________
  Hdf-forum is for HDF software users discussion.
  Hdf-forum@hdfgroup.org
  http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
  
--
Witold Eryk Wolski

Heidmark str 5
D-28329 Bremen
tel.: 04215261837