Dynamic variable size arrays in HDF5 Tables

Hi,

Could someone please give me an example of placing a dynamic variable
size array as a field column in an HDF5 table. I can't find any within
the documentation. I know I would have to create a new
datatype/dataspace for the column, but how do I create it to make it
dynamically resizable? Another way to do this is by creating a
separate table for the column type of fixed size and I could append as
many rows to it and then link it to the master table. How do I achieve
this link between the two tables? Is there an possibility to achieving
a automatic link, so during the read/write phase the master table
accesses the new column tables to pull/push data.

The example does not need to be in the HDF5_T api, as the table api is
just a synonym for the the compound datatypes.

Thanks.
Susheel

···

--
Susheel Varma
Computational Systems Biology | Tel : +44 (0) 114 22 25931
Department of Computer Science | Fax : +44 (0) 114 22 21810
The Kroto Research Institute | Web : www.dcs.shef.ac.uk/~susheel
University of Sheffield | Email: susheel.varma@sheffield.ac.uk
Broad Lane, Sheffield, S3 7HQ, UK | Email: susheel@dcs.shef.ac.uk

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Hi Susheel,

Hi,

Could someone please give me an example of placing a dynamic variable
size array as a field column in an HDF5 table. I can't find any within
the documentation. I know I would have to create a new
datatype/dataspace for the column, but how do I create it to make it
dynamically resizable? Another way to do this is by creating a
separate table for the column type of fixed size and I could append as
many rows to it and then link it to the master table. How do I achieve
this link between the two tables? Is there an possibility to achieving
a automatic link, so during the read/write phase the master table
accesses the new column tables to pull/push data.

  Try looking into the "variable-length" datatype information, they should be what you want.

  Quincey

···

On Jun 29, 2008, at 4:38 AM, Susheel Varma wrote:

The example does not need to be in the HDF5_T api, as the table api is
just a synonym for the the compound datatypes.

Thanks.
Susheel

--
Susheel Varma
Computational Systems Biology | Tel : +44 (0) 114 22 25931
Department of Computer Science | Fax : +44 (0) 114 22 21810
The Kroto Research Institute | Web : www.dcs.shef.ac.uk/~susheel
University of Sheffield | Email: susheel.varma@sheffield.ac.uk
Broad Lane, Sheffield, S3 7HQ, UK | Email: susheel@dcs.shef.ac.uk

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Thank you both for your valuable suggestions. Yes we try and avoid
using variable length arrays as much as possible, but some legacy code
tend to use them quite frequently. I'll try to do some performance
benchmarks before I start using it.

I was confused as the HDF5 Table API ref man say that "A table is
defined as a collection of records whose values are stored in
fixed-length fields"; fixed-length being the key here. I realize now
that Tables are just another abstraction on the compound datatype and
hence I'm guessing vlen datatypes will be supported within a table
too. Correct me if I'm wrong on any of this.

Thanks again.
Susheel

···

--
Susheel Varma
Computational Systems Biology | Tel : +44 (0) 114 22 25931
Department of Computer Science | Fax : +44 (0) 114 22 21810
The Kroto Research Institute | Web : www.dcs.shef.ac.uk/~susheel
University of Sheffield | Email: susheel.varma@sheffield.ac.uk
Broad Lane, Sheffield, S3 7HQ, UK | Email: susheel@dcs.shef.ac.uk

On Mon, Jun 30, 2008 at 2:13 PM, McCloskey, David L. <McClosDL@westinghouse.com> wrote:

Note that variable sized elements can be inefficient for large datasets because HDF5 stores the actual data on its "heap" space on the file, and only a reference to this data is contained within the actual dataset. Shouldn't be an issue for small data or infrequent access.

Dave McCloskey

-----Original Message-----
From: Susheel Varma [mailto:susheel.varma@gmail.com]
Sent: Sunday, June 29, 2008 5:38 AM
To: HDF Forum
Subject: [hdf-forum] Dynamic variable size arrays in HDF5 Tables

Hi,

Could someone please give me an example of placing a dynamic variable
size array as a field column in an HDF5 table. I can't find any within
the documentation. I know I would have to create a new
datatype/dataspace for the column, but how do I create it to make it
dynamically resizable? Another way to do this is by creating a
separate table for the column type of fixed size and I could append as
many rows to it and then link it to the master table. How do I achieve
this link between the two tables? Is there an possibility to achieving
a automatic link, so during the read/write phase the master table
accesses the new column tables to pull/push data.

The example does not need to be in the HDF5_T api, as the table api is
just a synonym for the the compound datatypes.

Thanks.
Susheel

--
Susheel Varma
Computational Systems Biology | Tel : +44 (0) 114 22 25931
Department of Computer Science | Fax : +44 (0) 114 22 21810
The Kroto Research Institute | Web : www.dcs.shef.ac.uk/~susheel
University of Sheffield | Email: susheel.varma@sheffield.ac.uk
Broad Lane, Sheffield, S3 7HQ, UK | Email: susheel@dcs.shef.ac.uk

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.