Can Dimension Scales be shared among datasets?

Looking at the dimension scale API

http://www.hdfgroup.org/HDF5/doc/HL/RM_H5DS.html
(btw., the pdf linked from this page is non-existent)

and the example program at

http://www.hdfgroup.org/HDF5/Tutor/h5dimscale.html

I wonder if it's possible to share the same dimension scale (i.e. its
datasets) between multiple datasets?

It won't be too uncommon to have many datasets with the same coordinates
attached. However, the dimension scale datasets seem to include a reference
to specific dataset, like an h5ls of the example HDF5 file shows:

/Xaxis Dataset {4/4}
    Attribute: CLASS scalar
        Type: 16-byte null-terminated ASCII string
        Data: "DIMENSION_SCALE"
    Attribute: REFERENCE_LIST {1}
        Type: struct {
                   "dataset" +0 object reference
                   "dimension" +8 native int
               } 16 bytes
    Location: 1:1672
    Links: 1
    Storage: 16 logical bytes, 16 allocated bytes, 100.00% utilization
    Type: native int

So it seems in this example the Xaxis dataset is bound to the dataset
DATASET-1:800 and another dataset with the same Xaxis would need to
replicate the same dimension scales.

Or can the dimension scale API handle sharing? It seems to be bound to
the dataspace, which cannot be shared among datasets (though I remember
there had been plans to implement such).

Another, related question - the DS API associates one dataset with
each dimension. Can it also be used to associate one dataset with *all*
dimensions? For instance, providing a two-dimensional dataset of (x,y)
coordinates as dimension scale for a two-dimensional dataset of data.

  Werner

···

Data: {DATASET-1:800, 1}

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

Thanks, Werner, for mentioning the missing PDF file; that link has
been fixed and the document should now be readily available.

Also, in case any readers of this forum had recently noticed that
the PDF copies of the "HDF5 Reference Manual" and the "HDF5 User's
Guide" were missing from the website, those files have been restored.

Regards,
-- Frank

···

At 22:58 -0500 2009-08-04, Werner Benger wrote:

Looking at the dimension scale API

http://www.hdfgroup.org/HDF5/doc/HL/RM_H5DS.html
(btw., the pdf linked from this page is non-existent)

Werner

That example creates a 2D dataset, and attaches one dimension scale to each one of its dimensions.

Every time you attach a dimension scale to a dataset, an attribute called "REFERENCE_LIST" is either created or updated on the dimension scale dataset. It contains an array of a compound type with 2 fields: a reference to the dataset that the dimension scale is attached to, and which dimension it is attached to.

So, this call in the example

/* attach the DS_1_NAME dimension scale to DSET_NAME at dimension index 0 */
if (H5DSattach_scale(did,dsid,DIM0)<0)
  goto out;

creates this attribute in the "Yaxis" dimension scale, shown here by h5dump (in the DATA section, 800 is the reference of the dataset "Mydata", and 0 is the dimension index it is attached to)

ATTRIBUTE "REFERENCE_LIST" {
         DATATYPE H5T_COMPOUND {
            H5T_REFERENCE "dataset";
            H5T_STD_I32LE "dimension";
         }
         DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
         DATA {
         (0): {
               DATASET 800 /Mydata ,
               0
            }
         }
      }

It is possible to share the same dimension scale between multiple datasets.

From that example, you would have to do something like

Create another dataset, here called DSET_NAME2

if (H5LTmake_dataset_int(fid,DSET_NAME2,rank,dims,buf)<0)
  goto out;

Open the IDs for the dataset DSET_NAME2 and the "Yaxis" dimension scale as shown in the example code and then attach them with

/* attach the DS_1_NAME dimension scale to DSET_NAME2 at dimension index 0 */
if (H5DSattach_scale(did,dsid,DIM0)<0)
  goto out;

If you open the generated file in h5dump , you'll see that "REFERENCE_LIST" has another entry to the new dataset "Mydata2"

ATTRIBUTE "REFERENCE_LIST" {
         DATATYPE H5T_COMPOUND {
            H5T_REFERENCE "dataset";
            H5T_STD_I32LE "dimension";
         }
         DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
         DATA {
         (0): {
               DATASET 800 /Mydata ,
               0
            },
         (1): {
               DATASET 1400 /Mydata2 ,
               0
            }
         }
      }

There is another attribute that is created or updated every time you call H5DSattach_scale. It is created on the dataset and it is called "DIMENSION_LIST". It has as many entries as the rank of the dataset (In this case, it has 2 entries because the dataset is 2D). Each entry is a variable length array of references. This is because it is also possible for a dimension of a dataset to be associated with *several* dimension scales. At each time, there is one "current" dimension scale associated, but it is possible to later choose another (to display in a visualization program, for example).

Here's the h5dump output of this attribute on the DATASET "Mydata"

ATTRIBUTE "DIMENSION_LIST" {
         DATATYPE H5T_VLEN { H5T_REFERENCE}
         DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
         DATA {
         (0): (DATASET 1672 /Yaxis ), (DATASET 4096 /Xaxis )
         }

Suppose you want to associate the index 0 of the dataset with another dimension scale. I modified the example with this code

Define a name for the new dimension scale

#define DS_3_NAME "Yaxis_other"

Define an array of values

float s3_wbuf[DIM1_SIZE] = {10,60,90}; /* data of DS 3 dataset */

Create a dataset for this dimension scale

/* make a DS dataset for the third dimension */
if (H5LTmake_dataset_float(fid,DS_3_NAME,rankds,s1_dim,s3_wbuf)<0)
  goto out;

and attach them as before, after getting the IDs (I send you the complete example as attached file).

Now, h5dump shows that index 0 of "DIMENSION_LIST" has a new entry

ATTRIBUTE "DIMENSION_LIST" {
         DATATYPE H5T_VLEN { H5T_REFERENCE}
         DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
         DATA {
         (0): (DATASET 1672 /Yaxis , DATASET 4368 /Yaxis_other ),
         (1): (DATASET 4096 /Xaxis )
         }

Now the fun part is to use a visualization program to see the differences in bitmaps generated from the dataset "Mydata"

HDF Explorer allows to choose between dimension scales if several are attached to one dimension index of a dataset.

HDF Explorer is available here

Open the file generated by the example, attached here. Right click on the dataset "Mydata" and choose "Discrete Map View". Note that the *first* dimension scale associated with index 0 of the dataset was used, the "Yaxis" dataset, which was defined as an array of

float s1_wbuf[DIM1_SIZE] = {10,20,30}; /* data of DS 1 dataset */

Now, you can choose to use "Yaxis_other" instead for the Y axis. Right click on the dataset "Mydata" and choose "Scale selection". A dialog box shows up

Click on the "dimensions" box, and you'll see that the "available dimension scales" box shows all dimensions that are associated with the *selected" index on the "dimensions" box. Here, there are two (HDF Explorer only shows the reference number). Choose the second and close the dialog box and do a new map as previously

Now the map is different, instead of the evenly spaced "Yaxis" dimension scale, we get the new "Yaxis_other" as a basis for the map generation (the map is just a translation of each value of the dataset into a color, the squares you see correspond each one to one data value).

Attached is the modified example and the hdf5 generated file.

Pedro

ex_ds3.h5 (9.27 KB)

ds_example.c (5.53 KB)

···

----- Original Message ----- From: "Werner Benger" <werner@cct.lsu.edu>
To: <hdf-forum@hdfgroup.org>
Sent: Tuesday, August 04, 2009 10:58 PM
Subject: [Hdf-forum] Can Dimension Scales be shared among datasets?

Looking at the dimension scale API

http://www.hdfgroup.org/HDF5/doc/HL/RM_H5DS.html
(btw., the pdf linked from this page is non-existent)

and the example program at

http://www.hdfgroup.org/HDF5/Tutor/h5dimscale.html

I wonder if it's possible to share the same dimension scale (i.e. its
datasets) between multiple datasets?

It won't be too uncommon to have many datasets with the same coordinates
attached. However, the dimension scale datasets seem to include a reference
to specific dataset, like an h5ls of the example HDF5 file shows:

/Xaxis Dataset {4/4}
   Attribute: CLASS scalar
       Type: 16-byte null-terminated ASCII string
       Data: "DIMENSION_SCALE"
   Attribute: REFERENCE_LIST {1}
       Type: struct {
                  "dataset" +0 object reference
                  "dimension" +8 native int
              } 16 bytes
       Data: {DATASET-1:800, 1}
   Location: 1:1672
   Links: 1
   Storage: 16 logical bytes, 16 allocated bytes, 100.00% utilization
   Type: native int

So it seems in this example the Xaxis dataset is bound to the dataset
DATASET-1:800 and another dataset with the same Xaxis would need to
replicate the same dimension scales.

Or can the dimension scale API handle sharing? It seems to be bound to
the dataspace, which cannot be shared among datasets (though I remember
there had been plans to implement such).

Another, related question - the DS API associates one dataset with
each dimension. Can it also be used to associate one dataset with *all*
dimensions? For instance, providing a two-dimensional dataset of (x,y)
coordinates as dimension scale for a two-dimensional dataset of data.

Werner

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Thanks, Frank, this document answers mostly my questions.

So it seems dimension scales *can* be shared (which I hope is also implemented
in the API and not just envisioned), and multidimensional dimension scales
can be used, but the behavior is undefined and "left to the application
to be interpreted".

I'm wondering if reference [12] "expanding raw data options" - formula
datasets and attributes - has seen light since then? The pdf states it
was to appear in 2005, is it out and available somewhere?

  Werner

···

On Wed, 05 Aug 2009 10:27:47 -0500, Frank Baker <fbaker@hdfgroup.org> wrote:

Thanks, Werner, for mentioning the missing PDF file; that link has
been fixed and the document should now be readily available.

Also, in case any readers of this forum had recently noticed that
the PDF copies of the "HDF5 Reference Manual" and the "HDF5 User's
Guide" were missing from the website, those files have been restored.

Regards,
-- Frank

At 22:58 -0500 2009-08-04, Werner Benger wrote:

Looking at the dimension scale API

http://www.hdfgroup.org/HDF5/doc/HL/RM_H5DS.html
(btw., the pdf linked from this page is non-existent)

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

Hi Werner

···

----- Original Message ----- From: "Werner Benger" <werner@cct.lsu.edu>
To: "Pedro Vicente" <pedro.vicente@space-research.org>
Sent: Friday, August 07, 2009 4:19 PM
Subject: Re: [Hdf-forum] Can Dimension Scales be shared among datasets?

Hi Pedro,

thanks for the valueable insight!

You're welcome

It seems the dimension scales correspond to "rectilinear" grids where the
coordinates in each dimension are given independently, but explictly.

Would there be a natural way to support a notion of "uniform" grids where
the coordinates are given equidistantly, by just specify origin and some
distance which is constant?

This idea was discussed but left out of the design. Dimension Scales have to be datasets.
To achieve this one would have to write a program that generates such a dataset (it would be possible to define an API function that does it, by specifying the origin and the distance).

Would there be a canonical way to associated coordinates with each of the
dataset elements ("curvilinear" grids), i.e. having one multidimensional,
multivalued dataset set associated as dimension scale to a dataset of the same
dimension? As far as I understand the design document, such is possible, but
undefined behavior with the interpretation left to the application.

Yes, possible, but the interpretation is left to the application.
Pedro

----------
Pedro Vicente
pedro.vicente@space-research.org
http://www.space-research.org/
Phone: (217) 898-8356
Mail: 1004 E. Mumford, Urbana, IL, 61801

Werner

On Fri, 07 Aug 2009 15:47:57 -0500, Pedro Vicente > <pedro.vicente@space-research.org> wrote:

and here is the image you get in HDF Explorer after choosing "Yaxis_other" as the basis for the map generation.

The array was defined as

float s3_wbuf[DIM1_SIZE] = {10,60,90}; /* data of DS 3 dataset */

so we get the uneven squares

Let me know if you have any questions
Thanks

Pedro

----------
Pedro Vicente
pedro.vicente@space-research.org
http://www.space-research.org/
Phone: (217) 898-8356
Mail: 1004 E. Mumford, Urbana, IL, 61801

----- Original Message -----
From: "Pedro Vicente" <pedro.vicente@space-research.org>
To: "Werner Benger" <werner@cct.lsu.edu>; <hdf-forum@hdfgroup.org>
Sent: Friday, August 07, 2009 3:26 PM
Subject: Re: [Hdf-forum] Can Dimension Scales be shared among datasets?

Werner

That example creates a 2D dataset, and attaches one dimension scale to each
one of its dimensions.

Every time you attach a dimension scale to a dataset, an attribute called
"REFERENCE_LIST" is either created or updated on the dimension scale
dataset. It contains an array of a compound type with 2 fields: a reference
to the dataset that the dimension scale is attached to, and which dimension
it is attached to.

So, this call in the example

/* attach the DS_1_NAME dimension scale to DSET_NAME at dimension index 0 */
if (H5DSattach_scale(did,dsid,DIM0)<0)
goto out;

creates this attribute in the "Yaxis" dimension scale, shown here by h5dump
(in the DATA section, 800 is the reference of the dataset "Mydata", and 0 is
the dimension index it is attached to)

ATTRIBUTE "REFERENCE_LIST" {
        DATATYPE H5T_COMPOUND {
           H5T_REFERENCE "dataset";
           H5T_STD_I32LE "dimension";
        }
        DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
        DATA {
        (0): {
              DATASET 800 /Mydata ,
              0
           }
        }
     }

It is possible to share the same dimension scale between multiple datasets.

From that example, you would have to do something like

Create another dataset, here called DSET_NAME2

if (H5LTmake_dataset_int(fid,DSET_NAME2,rank,dims,buf)<0)
goto out;

Open the IDs for the dataset DSET_NAME2 and the "Yaxis" dimension scale as
shown in the example code and then attach them with

/* attach the DS_1_NAME dimension scale to DSET_NAME2 at dimension index 0
*/
if (H5DSattach_scale(did,dsid,DIM0)<0)
goto out;

If you open the generated file in h5dump , you'll see that "REFERENCE_LIST"
has another entry to the new dataset "Mydata2"

ATTRIBUTE "REFERENCE_LIST" {
        DATATYPE H5T_COMPOUND {
           H5T_REFERENCE "dataset";
           H5T_STD_I32LE "dimension";
        }
        DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
        DATA {
        (0): {
              DATASET 800 /Mydata ,
              0
           },
        (1): {
              DATASET 1400 /Mydata2 ,
              0
           }
        }
     }

There is another attribute that is created or updated every time you call
H5DSattach_scale. It is created on the dataset and it is called
"DIMENSION_LIST". It has as many entries as the rank of the dataset (In this
case, it has 2 entries because the dataset is 2D). Each entry is a variable
length array of references. This is because it is also possible for a
dimension of a dataset to be associated with *several* dimension scales. At
each time, there is one "current" dimension scale associated, but it is
possible to later choose another (to display in a visualization program, for
example).

Here's the h5dump output of this attribute on the DATASET "Mydata"

ATTRIBUTE "DIMENSION_LIST" {
        DATATYPE H5T_VLEN { H5T_REFERENCE}
        DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
        DATA {
        (0): (DATASET 1672 /Yaxis ), (DATASET 4096 /Xaxis )
        }

Suppose you want to associate the index 0 of the dataset with another
dimension scale. I modified the example with this code

Define a name for the new dimension scale

#define DS_3_NAME "Yaxis_other"

Define an array of values

float s3_wbuf[DIM1_SIZE] = {10,60,90}; /* data of DS 3
dataset */

Create a dataset for this dimension scale

/* make a DS dataset for the third dimension */
if (H5LTmake_dataset_float(fid,DS_3_NAME,rankds,s1_dim,s3_wbuf)<0)
goto out;

and attach them as before, after getting the IDs (I send you the complete
example as attached file).

Now, h5dump shows that index 0 of "DIMENSION_LIST" has a new entry

ATTRIBUTE "DIMENSION_LIST" {
        DATATYPE H5T_VLEN { H5T_REFERENCE}
        DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
        DATA {
        (0): (DATASET 1672 /Yaxis , DATASET 4368 /Yaxis_other ),
        (1): (DATASET 4096 /Xaxis )
        }

Now the fun part is to use a visualization program to see the differences in
bitmaps generated from the dataset "Mydata"

HDF Explorer allows to choose between dimension scales if several are
attached to one dimension index of a dataset.

HDF Explorer is available here

http://www.space-research.org/

Open the file generated by the example, attached here. Right click on the
dataset "Mydata" and choose "Discrete Map View". Note that the *first*
dimension scale associated with index 0 of the dataset was used, the "Yaxis"
dataset, which was defined as an array of

float s1_wbuf[DIM1_SIZE] = {10,20,30}; /* data of DS 1
dataset */

Now, you can choose to use "Yaxis_other" instead for the Y axis. Right click
on the dataset "Mydata" and choose "Scale selection". A dialog box shows up

Click on the "dimensions" box, and you'll see that the "available dimension
scales" box shows all dimensions that are associated with the *selected"
index on the "dimensions" box. Here, there are two (HDF Explorer only shows
the reference number). Choose the second and close the dialog box and do a
new map as previously

Now the map is different, instead of the evenly spaced "Yaxis" dimension
scale, we get the new "Yaxis_other" as a basis for the map generation (the
map is just a translation of each value of the dataset into a color, the
squares you see correspond each one to one data value).

Attached is the modified example and the hdf5 generated file.

Pedro

----- Original Message -----
From: "Werner Benger" <werner@cct.lsu.edu>
To: <hdf-forum@hdfgroup.org>
Sent: Tuesday, August 04, 2009 10:58 PM
Subject: [Hdf-forum] Can Dimension Scales be shared among datasets?

Looking at the dimension scale API

http://www.hdfgroup.org/HDF5/doc/HL/RM_H5DS.html
(btw., the pdf linked from this page is non-existent)

and the example program at

http://www.hdfgroup.org/HDF5/Tutor/h5dimscale.html

I wonder if it's possible to share the same dimension scale (i.e. its
datasets) between multiple datasets?

It won't be too uncommon to have many datasets with the same coordinates
attached. However, the dimension scale datasets seem to include a
reference
to specific dataset, like an h5ls of the example HDF5 file shows:

/Xaxis Dataset {4/4}
   Attribute: CLASS scalar
       Type: 16-byte null-terminated ASCII string
       Data: "DIMENSION_SCALE"
   Attribute: REFERENCE_LIST {1}
       Type: struct {
                  "dataset" +0 object reference
                  "dimension" +8 native int
              } 16 bytes
       Data: {DATASET-1:800, 1}
   Location: 1:1672
   Links: 1
   Storage: 16 logical bytes, 16 allocated bytes, 100.00% utilization
   Type: native int

So it seems in this example the Xaxis dataset is bound to the dataset
DATASET-1:800 and another dataset with the same Xaxis would need to
replicate the same dimension scales.

Or can the dimension scale API handle sharing? It seems to be bound to
the dataspace, which cannot be shared among datasets (though I remember
there had been plans to implement such).

Another, related question - the DS API associates one dataset with
each dimension. Can it also be used to associate one dataset with *all*
dimensions? For instance, providing a two-dimensional dataset of (x,y)
coordinates as dimension scale for a two-dimensional dataset of data.

Werner

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization
Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--------------------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

Werner

Here is the example of how Dimension Scales can be chosen by HDF Explorer.

Opening the file generated by the example, that I sent previously, right clicking on the dataset "Mydata" and choosing "Discrete Map View", we get

Note that the *first* dimension scale associated with index 0 of the dataset was used, the "Yaxis" dataset, which was defined as an array of
float s1_wbuf[DIM1_SIZE] = {10,20,30}; /* data of DS 1 dataset */

After choosing to use "Yaxis_other" instead for the Y axis on the dialog box that shows up in the "Scale selection" menu, we get.

This was the array that was unevenly spaced defined as

float s3_wbuf[DIM1_SIZE] = {10,60,90}; /* data of DS 3 dataset */

Each square on the images corresponds to a dataset index location.

Pedro

···

----- Original Message -----
From: "Pedro Vicente" <pedro.vicente@space-research.org>
To: "Werner Benger" <werner@cct.lsu.edu>; <hdf-forum@hdfgroup.org>
Sent: Friday, August 07, 2009 3:26 PM
Subject: Re: [Hdf-forum] Can Dimension Scales be shared among datasets?

Werner

That example creates a 2D dataset, and attaches one dimension scale to each
one of its dimensions.

Every time you attach a dimension scale to a dataset, an attribute called
"REFERENCE_LIST" is either created or updated on the dimension scale
dataset. It contains an array of a compound type with 2 fields: a reference
to the dataset that the dimension scale is attached to, and which dimension
it is attached to.

So, this call in the example

/* attach the DS_1_NAME dimension scale to DSET_NAME at dimension index 0 */
if (H5DSattach_scale(did,dsid,DIM0)<0)
goto out;

creates this attribute in the "Yaxis" dimension scale, shown here by h5dump
(in the DATA section, 800 is the reference of the dataset "Mydata", and 0 is
the dimension index it is attached to)

ATTRIBUTE "REFERENCE_LIST" {
        DATATYPE H5T_COMPOUND {
           H5T_REFERENCE "dataset";
           H5T_STD_I32LE "dimension";
        }
        DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
        DATA {
        (0): {
              DATASET 800 /Mydata ,
              0
           }
        }
     }

It is possible to share the same dimension scale between multiple datasets.

From that example, you would have to do something like

Create another dataset, here called DSET_NAME2

if (H5LTmake_dataset_int(fid,DSET_NAME2,rank,dims,buf)<0)
goto out;

Open the IDs for the dataset DSET_NAME2 and the "Yaxis" dimension scale as
shown in the example code and then attach them with

/* attach the DS_1_NAME dimension scale to DSET_NAME2 at dimension index 0
*/
if (H5DSattach_scale(did,dsid,DIM0)<0)
goto out;

If you open the generated file in h5dump , you'll see that "REFERENCE_LIST"
has another entry to the new dataset "Mydata2"

ATTRIBUTE "REFERENCE_LIST" {
        DATATYPE H5T_COMPOUND {
           H5T_REFERENCE "dataset";
           H5T_STD_I32LE "dimension";
        }
        DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
        DATA {
        (0): {
              DATASET 800 /Mydata ,
              0
           },
        (1): {
              DATASET 1400 /Mydata2 ,
              0
           }
        }
     }

There is another attribute that is created or updated every time you call
H5DSattach_scale. It is created on the dataset and it is called
"DIMENSION_LIST". It has as many entries as the rank of the dataset (In this
case, it has 2 entries because the dataset is 2D). Each entry is a variable
length array of references. This is because it is also possible for a
dimension of a dataset to be associated with *several* dimension scales. At
each time, there is one "current" dimension scale associated, but it is
possible to later choose another (to display in a visualization program, for
example).

Here's the h5dump output of this attribute on the DATASET "Mydata"

ATTRIBUTE "DIMENSION_LIST" {
        DATATYPE H5T_VLEN { H5T_REFERENCE}
        DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
        DATA {
        (0): (DATASET 1672 /Yaxis ), (DATASET 4096 /Xaxis )
        }

Suppose you want to associate the index 0 of the dataset with another
dimension scale. I modified the example with this code

Define a name for the new dimension scale

#define DS_3_NAME "Yaxis_other"

Define an array of values

float s3_wbuf[DIM1_SIZE] = {10,60,90}; /* data of DS 3
dataset */

Create a dataset for this dimension scale

/* make a DS dataset for the third dimension */
if (H5LTmake_dataset_float(fid,DS_3_NAME,rankds,s1_dim,s3_wbuf)<0)
goto out;

and attach them as before, after getting the IDs (I send you the complete
example as attached file).

Now, h5dump shows that index 0 of "DIMENSION_LIST" has a new entry

ATTRIBUTE "DIMENSION_LIST" {
        DATATYPE H5T_VLEN { H5T_REFERENCE}
        DATASPACE SIMPLE { ( 2 ) / ( 2 ) }
        DATA {
        (0): (DATASET 1672 /Yaxis , DATASET 4368 /Yaxis_other ),
        (1): (DATASET 4096 /Xaxis )
        }

Now the fun part is to use a visualization program to see the differences in
bitmaps generated from the dataset "Mydata"

HDF Explorer allows to choose between dimension scales if several are
attached to one dimension index of a dataset.

HDF Explorer is available here

http://www.space-research.org/

Open the file generated by the example, attached here. Right click on the
dataset "Mydata" and choose "Discrete Map View". Note that the *first*
dimension scale associated with index 0 of the dataset was used, the "Yaxis"
dataset, which was defined as an array of

float s1_wbuf[DIM1_SIZE] = {10,20,30}; /* data of DS 1
dataset */

Now, you can choose to use "Yaxis_other" instead for the Y axis. Right click
on the dataset "Mydata" and choose "Scale selection". A dialog box shows up

Click on the "dimensions" box, and you'll see that the "available dimension
scales" box shows all dimensions that are associated with the *selected"
index on the "dimensions" box. Here, there are two (HDF Explorer only shows
the reference number). Choose the second and close the dialog box and do a
new map as previously

Now the map is different, instead of the evenly spaced "Yaxis" dimension
scale, we get the new "Yaxis_other" as a basis for the map generation (the
map is just a translation of each value of the dataset into a color, the
squares you see correspond each one to one data value).

Attached is the modified example and the hdf5 generated file.

Pedro

----- Original Message -----
From: "Werner Benger" <werner@cct.lsu.edu>
To: <hdf-forum@hdfgroup.org>
Sent: Tuesday, August 04, 2009 10:58 PM
Subject: [Hdf-forum] Can Dimension Scales be shared among datasets?

Looking at the dimension scale API

http://www.hdfgroup.org/HDF5/doc/HL/RM_H5DS.html
(btw., the pdf linked from this page is non-existent)

and the example program at

http://www.hdfgroup.org/HDF5/Tutor/h5dimscale.html

I wonder if it's possible to share the same dimension scale (i.e. its
datasets) between multiple datasets?

It won't be too uncommon to have many datasets with the same coordinates
attached. However, the dimension scale datasets seem to include a
reference
to specific dataset, like an h5ls of the example HDF5 file shows:

/Xaxis Dataset {4/4}
   Attribute: CLASS scalar
       Type: 16-byte null-terminated ASCII string
       Data: "DIMENSION_SCALE"
   Attribute: REFERENCE_LIST {1}
       Type: struct {
                  "dataset" +0 object reference
                  "dimension" +8 native int
              } 16 bytes
       Data: {DATASET-1:800, 1}
   Location: 1:1672
   Links: 1
   Storage: 16 logical bytes, 16 allocated bytes, 100.00% utilization
   Type: native int

So it seems in this example the Xaxis dataset is bound to the dataset
DATASET-1:800 and another dataset with the same Xaxis would need to
replicate the same dimension scales.

Or can the dimension scale API handle sharing? It seems to be bound to
the dataspace, which cannot be shared among datasets (though I remember
there had been plans to implement such).

Another, related question - the DS API associates one dataset with
each dimension. Can it also be used to associate one dataset with *all*
dimensions? For instance, providing a two-dimensional dataset of (x,y)
coordinates as dimension scale for a two-dimensional dataset of data.

Werner

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization
Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--------------------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Thanks, Frank, this document answers mostly my questions.

So it seems dimension scales *can* be shared (which I hope is also implemented
in the API and not just envisioned), and multidimensional dimension scales
can be used, but the behavior is undefined and "left to the application
to be interpreted".

I'm wondering if reference [12] "expanding raw data options" - formula
datasets and attributes - has seen light since then? The pdf states it
was to appear in 2005, is it out and available somewhere?

Formula datasets and attributes are still on our to-do list ;-(

Elena

···

On Aug 7, 2009, at 12:45 AM, Werner Benger wrote:

  Werner

On Wed, 05 Aug 2009 10:27:47 -0500, Frank Baker > <fbaker@hdfgroup.org> wrote:

Thanks, Werner, for mentioning the missing PDF file; that link has
been fixed and the document should now be readily available.

Also, in case any readers of this forum had recently noticed that
the PDF copies of the "HDF5 Reference Manual" and the "HDF5 User's
Guide" were missing from the website, those files have been restored.

Regards,
-- Frank

At 22:58 -0500 2009-08-04, Werner Benger wrote:

Looking at the dimension scale API

http://www.hdfgroup.org/HDF5/doc/HL/RM_H5DS.html
(btw., the pdf linked from this page is non-existent)

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

I'm wondering if reference [12] "expanding raw data options" - formula
datasets and attributes - has seen light since then? The pdf states it
was to appear in 2005, is it out and available somewhere?

Formula datasets and attributes are still on our to-do list ;-(

Elena

Do you have any information on how this was/is planned?

I need this anyway and would therefore like do it in a direction that
is close/compatible to HDF5's ideas.

   Werner

···

On Fri, 07 Aug 2009 08:59:28 -0500, Elena Pourmal <epourmal@hdfgroup.org> wrote:

On Wed, 05 Aug 2009 10:27:47 -0500, Frank Baker >> <fbaker@hdfgroup.org> wrote:

Thanks, Werner, for mentioning the missing PDF file; that link has
been fixed and the document should now be readily available.

Also, in case any readers of this forum had recently noticed that
the PDF copies of the "HDF5 Reference Manual" and the "HDF5 User's
Guide" were missing from the website, those files have been restored.

Regards,
-- Frank

At 22:58 -0500 2009-08-04, Werner Benger wrote:

Looking at the dimension scale API

http://www.hdfgroup.org/HDF5/doc/HL/RM_H5DS.html
(btw., the pdf linked from this page is non-existent)

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization
Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

Hi Werner,

I'm wondering if reference [12] "expanding raw data options" - formula
datasets and attributes - has seen light since then? The pdf states it
was to appear in 2005, is it out and available somewhere?

Formula datasets and attributes are still on our to-do list ;-(

Elena

Do you have any information on how this was/is planned?

I need this anyway and would therefore like do it in a direction that
is close/compatible to HDF5's ideas.

  I'd like to make it part of the internal metadata for a dataset, similar to choosing the chunked/contiguous/compact layout for a dataset. That being the case, I think it's reasonable to code as a wrapper around HDF5 and store the information you are interested in as an attribute on a dataset with "late" allocation time. Then, you'd have to use wrapper calls to "read" from the formula dataset (and issue errors on writes?)

  Quincey

···

On Aug 7, 2009, at 10:57 AM, Werner Benger wrote:

On Fri, 07 Aug 2009 08:59:28 -0500, Elena Pourmal <epourmal@hdfgroup.org > > wrote:

  Werner

On Wed, 05 Aug 2009 10:27:47 -0500, Frank Baker >>> <fbaker@hdfgroup.org> wrote:

Thanks, Werner, for mentioning the missing PDF file; that link has
been fixed and the document should now be readily available.

Also, in case any readers of this forum had recently noticed that
the PDF copies of the "HDF5 Reference Manual" and the "HDF5 User's
Guide" were missing from the website, those files have been restored.

Regards,
-- Frank

At 22:58 -0500 2009-08-04, Werner Benger wrote:

Looking at the dimension scale API

http://www.hdfgroup.org/HDF5/doc/HL/RM_H5DS.html
(btw., the pdf linked from this page is non-existent)

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization
Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Hi Quincey,

Hi Werner,

I'm wondering if reference [12] "expanding raw data options" -
formula
datasets and attributes - has seen light since then? The pdf
states it
was to appear in 2005, is it out and available somewhere?

Formula datasets and attributes are still on our to-do list ;-(

Elena

Do you have any information on how this was/is planned?

I need this anyway and would therefore like do it in a direction that
is close/compatible to HDF5's ideas.

  I'd like to make it part of the internal metadata for a dataset,
similar to choosing the chunked/contiguous/compact layout for a
dataset. That being the case, I think it's reasonable to code as a
wrapper around HDF5 and store the information you are interested in as
an attribute on a dataset with "late" allocation time. Then, you'd
have to use wrapper calls to "read" from the formula dataset (and
issue errors on writes?)

How would you specify a formula? As some arbitrary ascii (or utf8) string
that has to be parsed by the application, or e.g. some tree of instructions
which is an already parsed formula that only has to be evaluated by the
application at runtime and data access?

I'm for now just interested in a polynomial expression, where basically
the coefficients need to be stored - possibly as part of this formula,
or eventually in a dataset itself.

Can you provide some example how such could look like?

In theory, a formula could be stored as a parsed tree

/formula/ (group)
/formula/A (dataset)
/formula/+/ (group)
/formula/+/B (dataset)
/formula/+/*/ (group)
/formula/+/C (dataset)

which could mean formula = A + (B*C) , but I guess you have
something different mind such that it can look like the
layout of a dataset?

  Werner

···

On Fri, 07 Aug 2009 16:02:27 -0500, Quincey Koziol <koziol@hdfgroup.org> wrote:

On Aug 7, 2009, at 10:57 AM, Werner Benger wrote:

On Fri, 07 Aug 2009 08:59:28 -0500, Elena Pourmal <epourmal@hdfgroup.org >> > wrote:

  Quincey

  Werner

On Wed, 05 Aug 2009 10:27:47 -0500, Frank Baker >>>> <fbaker@hdfgroup.org> wrote:

Thanks, Werner, for mentioning the missing PDF file; that link has
been fixed and the document should now be readily available.

Also, in case any readers of this forum had recently noticed that
the PDF copies of the "HDF5 Reference Manual" and the "HDF5 User's
Guide" were missing from the website, those files have been
restored.

Regards,
-- Frank

At 22:58 -0500 2009-08-04, Werner Benger wrote:

Looking at the dimension scale API

http://www.hdfgroup.org/HDF5/doc/HL/RM_H5DS.html
(btw., the pdf linked from this page is non-existent)

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization
Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization
Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

Hi Werner,

Hi Quincey,

Hi Werner,

I'm wondering if reference [12] "expanding raw data options" -
formula
datasets and attributes - has seen light since then? The pdf
states it
was to appear in 2005, is it out and available somewhere?

Formula datasets and attributes are still on our to-do list ;-(

Elena

Do you have any information on how this was/is planned?

I need this anyway and would therefore like do it in a direction that
is close/compatible to HDF5's ideas.

  I'd like to make it part of the internal metadata for a dataset,
similar to choosing the chunked/contiguous/compact layout for a
dataset. That being the case, I think it's reasonable to code as a
wrapper around HDF5 and store the information you are interested in as
an attribute on a dataset with "late" allocation time. Then, you'd
have to use wrapper calls to "read" from the formula dataset (and
issue errors on writes?)

How would you specify a formula? As some arbitrary ascii (or utf8) string
that has to be parsed by the application, or e.g. some tree of instructions
which is an already parsed formula that only has to be evaluated by the
application at runtime and data access?

  Hmm, this is one of the reasons we haven't done it yet - there's lots of things still to decide. :slight_smile: I would probably just lean toward a simple text string which had placeholders for the coordinates of the dataset. Something like "<0> * 3 + <1>", where <0> was the coordinate in the slowest changing dimension and <1> was the next fastest changing etc.

I'm for now just interested in a polynomial expression, where basically
the coefficients need to be stored - possibly as part of this formula,
or eventually in a dataset itself.

Can you provide some example how such could look like?

In theory, a formula could be stored as a parsed tree

/formula/ (group)
/formula/A (dataset)
/formula/+/ (group)
/formula/+/B (dataset)
/formula/+/*/ (group)
/formula/+/C (dataset)

which could mean formula = A + (B*C) , but I guess you have
something different mind such that it can look like the
layout of a dataset?

  Ah, you are composing one dataset out of formulas that involve other datasets, not just basing the formula on the coordinates within the dataset. I suppose you could expand your 'language' for expressing the formula to encompass that also. I think it would be best to think about all the use cases you wanted to handle and then write a BNF description of the language which will handle them.

  Quincey

···

On Aug 7, 2009, at 4:14 PM, Werner Benger wrote:

On Fri, 07 Aug 2009 16:02:27 -0500, Quincey Koziol <koziol@hdfgroup.org > > wrote:

On Aug 7, 2009, at 10:57 AM, Werner Benger wrote:

On Fri, 07 Aug 2009 08:59:28 -0500, Elena Pourmal <epourmal@hdfgroup.org >>>> wrote:

  Werner

  Quincey

  Werner

On Wed, 05 Aug 2009 10:27:47 -0500, Frank Baker >>>>> <fbaker@hdfgroup.org> wrote:

Thanks, Werner, for mentioning the missing PDF file; that link has
been fixed and the document should now be readily available.

Also, in case any readers of this forum had recently noticed that
the PDF copies of the "HDF5 Reference Manual" and the "HDF5 User's
Guide" were missing from the website, those files have been
restored.

Regards,
-- Frank

At 22:58 -0500 2009-08-04, Werner Benger wrote:

Looking at the dimension scale API

http://www.hdfgroup.org/HDF5/doc/HL/RM_H5DS.html
(btw., the pdf linked from this page is non-existent)

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization
Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization
Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Hi Quincey,

Hi Werner,

Hi Quincey,

Hi Werner,

I'm wondering if reference [12] "expanding raw data options" -
formula
datasets and attributes - has seen light since then? The pdf
states it
was to appear in 2005, is it out and available somewhere?

Formula datasets and attributes are still on our to-do list ;-(

Elena

Do you have any information on how this was/is planned?

I need this anyway and would therefore like do it in a direction
that
is close/compatible to HDF5's ideas.

  I'd like to make it part of the internal metadata for a dataset,
similar to choosing the chunked/contiguous/compact layout for a
dataset. That being the case, I think it's reasonable to code as a
wrapper around HDF5 and store the information you are interested in
as
an attribute on a dataset with "late" allocation time. Then, you'd
have to use wrapper calls to "read" from the formula dataset (and
issue errors on writes?)

How would you specify a formula? As some arbitrary ascii (or utf8)
string
that has to be parsed by the application, or e.g. some tree of
instructions
which is an already parsed formula that only has to be evaluated by
the
application at runtime and data access?

  Hmm, this is one of the reasons we haven't done it yet - there's lots
of things still to decide. :slight_smile: I would probably just lean toward a
simple text string which had placeholders for the coordinates of the
dataset. Something like "<0> * 3 + <1>", where <0> was the coordinate
in the slowest changing dimension and <1> was the next fastest
changing etc.

Yes, certainly an interesting topic to consider. I'd lean against a simple
text string but rather some way to reference existing attributes and datasets;
without a text string you don't need to define parsing rules, but already
have something which "only needs to be evaluated". Plus, it may refer to
datasets that are already in binary format - doubles, floats, etc., otherwise
there will be the problem of parsing some text string which contains a floating
point number with all the precision loss and platform dependency that might
occur there. There are already mechanisms in HDF5 to directly reference another
dataset, like the dimension scales also do; would be nice to juse re-use such,
instead of adding some string parser to either the HDF5 library, or requiring
the applications to implement such. However, how to make such invisible as
a dataset layout, I don't know. I guess the idea is that an application could
read a "procedural dataset" as if it were an explicitly stored dataset, with
the formula evaluator being build invisibly into HDF5? Technically I could
imagine that such could be implemented right now via some compression filter
as well, with the compression containing the formula evaluator.

I'm for now just interested in a polynomial expression, where
basically
the coefficients need to be stored - possibly as part of this formula,
or eventually in a dataset itself.

Can you provide some example how such could look like?

In theory, a formula could be stored as a parsed tree

/formula/ (group)
/formula/A (dataset)
/formula/+/ (group)
/formula/+/B (dataset)
/formula/+/*/ (group)
/formula/+/C (dataset)

which could mean formula = A + (B*C) , but I guess you have
something different mind such that it can look like the
layout of a dataset?

  Ah, you are composing one dataset out of formulas that involve other
datasets, not just basing the formula on the coordinates within the
dataset. I suppose you could expand your 'language' for expressing
the formula to encompass that also. I think it would be best to think
about all the use cases you wanted to handle and then write a BNF
description of the language which will handle them.

Well one of these datasets, like "B", could be the coordinates (or indices)
of some dataset itself, of course. However, the main idea was to not have a language
at all - but to store the tree of objects that need to be evaluated, thereby
skipping the step of being required to parse a string conforming to some
language definition. There are still some open questions of course, like
what should happen if the datasets in the above formula tree are not of
the same dataspace (share dataspaces would really be helpful here), and
the algebraic operation of each operator would need to be defined somewhere
(for +-*/ it would be straightforward,but %,^,& would require some conventions).

  Werner

···

On Fri, 07 Aug 2009 17:02:52 -0500, Quincey Koziol <koziol@hdfgroup.org> wrote:

On Aug 7, 2009, at 4:14 PM, Werner Benger wrote:

On Fri, 07 Aug 2009 16:02:27 -0500, Quincey Koziol <koziol@hdfgroup.org >> > wrote:

On Aug 7, 2009, at 10:57 AM, Werner Benger wrote:

On Fri, 07 Aug 2009 08:59:28 -0500, Elena Pourmal <epourmal@hdfgroup.org >>>>> wrote:

  Quincey

  Werner

  Quincey

  Werner

On Wed, 05 Aug 2009 10:27:47 -0500, Frank Baker >>>>>> <fbaker@hdfgroup.org> wrote:

Thanks, Werner, for mentioning the missing PDF file; that link
has
been fixed and the document should now be readily available.

Also, in case any readers of this forum had recently noticed that
the PDF copies of the "HDF5 Reference Manual" and the "HDF5
User's
Guide" were missing from the website, those files have been
restored.

Regards,
-- Frank

At 22:58 -0500 2009-08-04, Werner Benger wrote:

Looking at the dimension scale API

http://www.hdfgroup.org/HDF5/doc/HL/RM_H5DS.html
(btw., the pdf linked from this page is non-existent)

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization
Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization
Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization
Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362