Newbie questions regarding tables

HI Folks,

I am an HDF5 newbie, so please be patient. I have a few basic
questions after reading the tutorials and such. First, I will say that
I have telemetry data (timestamp with state information) which I need
to stream to an HDF file. The data nicely fits the model of a table.

My Sys Admin has installed HDF5 version 1.9.131.

My first question is regard to your table APIs. I see that an HDF
file can consist of the fundamental constructs of group and/or dataset.
However, it seems the table and packetTable are also fundamental in
that I do not have to package either of these tables inside of a group to
make a valid HDF file. (1) Is this the way these are intended to be used?

My problem with the regular table is that during its creation, it does
not hand back an identifier so I cannot attach attributes directly to
the table. I would like the columns in the table to have "units" and
a "description" as meta-data for the table. I can put the regular
table and an attributes into a group (where the attributes
are attached to the group) which seems not too bad overall.
(2) Is this what one would ordinarily do for this issue?
(3) Maybe I made a mistake, but I cannot h5dump or h5stat the file I
have generated with a regular table. Is this expected?

(4) As for the packet table, I can attach attributes, but the packet table
does not really have the concept of columns as far as I can tell. So
is it assumed that meta-data will be handled in the attributes dataset
which can attach to the packet table?

Is it that I am just to bound by my conventional view of the world, and
HDF is more general. Can I make the packetTable actually anything
I want it to be via attributes?

Thanks,
Michele

···

--
Michele D. De La Pena Scientific Software Engineer
Large Binocular Telescope Obs. email: mdelapena@lbto.org
933 N. Cherry Avenue phone: 520-626-1402
Tucson, AZ 85721-0065 cell: 520-419-8825

Michele,

I'm dealing with a similar timestamped data streams and use Packet Tables almost exclusively.

My solution for the unit labels problem was to attach an attribute to the Packet Table. The Unit attribute is an array of strings, one per column.

Another important trick, for HDF5.net at least, you need to open the Packet Table as a (H5D) dataset to attach/read an attribute.

I'm not sure what you mean by (4) Packet Tables not having the concept of a column?

Scott

···

From: Hdf-forum [mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Michele De La Pena
Sent: Monday, October 22, 2012 5:54 PM
To: hdf-forum@hdfgroup.org
Subject: [Hdf-forum] Newbie questions regarding tables

HI Folks,

I am an HDF5 newbie, so please be patient. I have a few basic
questions after reading the tutorials and such. First, I will say that
I have telemetry data (timestamp with state information) which I need
to stream to an HDF file. The data nicely fits the model of a table.

My Sys Admin has installed HDF5 version 1.9.131.

My first question is regard to your table APIs. I see that an HDF
file can consist of the fundamental constructs of group and/or dataset.
However, it seems the table and packetTable are also fundamental in
that I do not have to package either of these tables inside of a group to
make a valid HDF file. (1) Is this the way these are intended to be used?

My problem with the regular table is that during its creation, it does
not hand back an identifier so I cannot attach attributes directly to
the table. I would like the columns in the table to have "units" and
a "description" as meta-data for the table. I can put the regular
table and an attributes into a group (where the attributes
are attached to the group) which seems not too bad overall.
(2) Is this what one would ordinarily do for this issue?
(3) Maybe I made a mistake, but I cannot h5dump or h5stat the file I
have generated with a regular table. Is this expected?

(4) As for the packet table, I can attach attributes, but the packet table
does not really have the concept of columns as far as I can tell. So
is it assumed that meta-data will be handled in the attributes dataset
which can attach to the packet table?

Is it that I am just to bound by my conventional view of the world, and
HDF is more general. Can I make the packetTable actually anything
I want it to be via attributes?

Thanks,
Michele

--
Michele D. De La Pena Scientific Software Engineer
Large Binocular Telescope Obs. email: mdelapena@lbto.org<mailto:mdelapena@lbto.org>
933 N. Cherry Avenue phone: 520-626-1402
Tucson, AZ 85721-0065 cell: 520-419-8825

________________________________

Email addresses of ITT Exelis employees have changed from itt.com to exelisinc.com. Please update your favorites and contact information to reflect these changes.

This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Exelis Inc. The recipient should check this e-mail and any attachments for the presence of viruses. Exelis Inc. accepts no liability for any damage caused by any virus transmitted by this e-mail.

HI,
Thanks for the input.

Michele,

I’m dealing with a similar timestamped data streams and use Packet Tables
almost exclusively.

After some more thought, I decided this was probably the better way to go.
I can append data as needed.

My solution for the unit labels problem was to attach an attribute to the
Packet Table. The Unit attribute is an array of strings, one per column.

Indeed, it looks like I will need to do this for "column" names, data
types, descriptions, etc.

Another important trick, for HDF5.net at least, you need to open the
Packet Table as a (H5D) dataset to attach/read an attribute.

Thanks. I have been out of the office for a few days, so I will be doing
more today.

I’m not sure what you mean by (4) Packet Tables not having the concept of
a column?

The "regular" table has field names, field types, etc. This table seemed
to lend itself more to the table
concept. However, I can see (hope I can learn to use) the power of the
attribute.

I will send an update when I have success (!) or need help.
Thanks,
Michele

···

On Tue, Oct 23, 2012 at 7:41 AM, Mitchell, Scott - IS < Scott.Mitchell@exelisinc.com> wrote:

Scott

*From:* Hdf-forum [mailto:hdf-forum-bounces@hdfgroup.org] *On Behalf Of *Michele
De La Pena
*Sent:* Monday, October 22, 2012 5:54 PM
*To:* hdf-forum@hdfgroup.org
*Subject:* [Hdf-forum] Newbie questions regarding tables

HI Folks,

I am an HDF5 newbie, so please be patient. I have a few basic
questions after reading the tutorials and such. First, I will say that
I have telemetry data (timestamp with state information) which I need
to stream to an HDF file. The data nicely fits the model of a table.

My Sys Admin has installed HDF5 version 1.9.131.

My first question is regard to your table APIs. I see that an HDF
file can consist of the fundamental constructs of group and/or dataset.
However, it seems the table and packetTable are also fundamental in
that I do not have to package either of these tables inside of a group to
make a valid HDF file. (1) Is this the way these are intended to be used?

My problem with the regular table is that during its creation, it does
not hand back an identifier so I cannot attach attributes directly to
the table. I would like the columns in the table to have "units" and
a "description" as meta-data for the table. I can put the regular
table and an attributes into a group (where the attributes
are attached to the group) which seems not too bad overall.
(2) Is this what one would ordinarily do for this issue?
(3) Maybe I made a mistake, but I cannot h5dump or h5stat the file I
have generated with a regular table. Is this expected?

(4) As for the packet table, I can attach attributes, but the packet table
does not really have the concept of columns as far as I can tell. So
is it assumed that meta-data will be handled in the attributes dataset
which can attach to the packet table?

Is it that I am just to bound by my conventional view of the world, and
HDF is more general. Can I make the packetTable actually anything
I want it to be via attributes?

Thanks,
Michele

--
Michele D. De La Pena Scientific Software Engineer
Large Binocular Telescope Obs. email: mdelapena@lbto.org
933 N. Cherry Avenue phone: 520-626-1402
Tucson, AZ 85721-0065 cell: 520-419-8825

------------------------------

Email addresses of ITT Exelis employees have changed from itt.com to
exelisinc.com. Please update your favorites and contact information to
reflect these changes.

This e-mail and any files transmitted with it may be proprietary and are
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this e-mail in error please notify the
sender. Please note that any views or opinions presented in this e-mail are
solely those of the author and do not necessarily represent those of Exelis
Inc. The recipient should check this e-mail and any attachments for the
presence of viruses. Exelis Inc. accepts no liability for any damage caused
by any virus transmitted by this e-mail.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Michele D. De La Pena Scientific Software Engineer
Large Binocular Telescope Obs. email: mdelapena@lbto.org
933 N. Cherry Avenue phone: 520-626-1402
Tucson, AZ 85721-0065 cell: 520-419-8825

HI Again,

I guess I need a pointer from you. I have created by first HDF5 file, and
it contains a packet table (yeah).
When I tried to attach attributes, I got an error

HDF5-DIAG: Error detected in HDF5 (1.9.131) thread 0:
  #000: H5A.c line 242 in H5Acreate2(): not a location
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: H5Gloc.c line 253 in H5G_loc(): invalid object ID
    major: Invalid arguments to routine
    minor: Bad value

From my searching on the Web and your kind warning below (I fell into the

hole anyway), I see
what you mean. However, I am not exactly sure of your solution. Do you
mean while I created
the packet table as

hid_t ptTable = H5PTcreate_fl (fileID, "PCS Weather Table", memType,
chunkSize, compress);

that I need to close the table, and then open the table as a dataset as
below?

hid_t ptAsDs = H5Dopen (ptTable, "PCS Weather Table", H5P_DEFAULT);

Use of ptTable does not work in this open statement. I tried using the
file identifier in the
H5Dopen in conjunction with the packet table name, and this seems to work
(I have other
errors). Is this the accepted way to do this?

Thanks,
Michele

···

On Tue, Oct 23, 2012 at 7:41 AM, Mitchell, Scott - IS < Scott.Mitchell@exelisinc.com> wrote:

Michele,

I’m dealing with a similar timestamped data streams and use Packet Tables
almost exclusively.

My solution for the unit labels problem was to attach an attribute to the
Packet Table. The Unit attribute is an array of strings, one per column.

Another important trick, for HDF5.net at least, you need to open the
Packet Table as a (H5D) dataset to attach/read an attribute.

I’m not sure what you mean by (4) Packet Tables not having the concept of
a column?

Scott

*From:* Hdf-forum [mailto:hdf-forum-bounces@hdfgroup.org] *On Behalf Of *Michele
De La Pena
*Sent:* Monday, October 22, 2012 5:54 PM
*To:* hdf-forum@hdfgroup.org
*Subject:* [Hdf-forum] Newbie questions regarding tables

HI Folks,

I am an HDF5 newbie, so please be patient. I have a few basic
questions after reading the tutorials and such. First, I will say that
I have telemetry data (timestamp with state information) which I need
to stream to an HDF file. The data nicely fits the model of a table.

My Sys Admin has installed HDF5 version 1.9.131.

My first question is regard to your table APIs. I see that an HDF
file can consist of the fundamental constructs of group and/or dataset.
However, it seems the table and packetTable are also fundamental in
that I do not have to package either of these tables inside of a group to
make a valid HDF file. (1) Is this the way these are intended to be used?

My problem with the regular table is that during its creation, it does
not hand back an identifier so I cannot attach attributes directly to
the table. I would like the columns in the table to have "units" and
a "description" as meta-data for the table. I can put the regular
table and an attributes into a group (where the attributes
are attached to the group) which seems not too bad overall.
(2) Is this what one would ordinarily do for this issue?
(3) Maybe I made a mistake, but I cannot h5dump or h5stat the file I
have generated with a regular table. Is this expected?

(4) As for the packet table, I can attach attributes, but the packet table
does not really have the concept of columns as far as I can tell. So
is it assumed that meta-data will be handled in the attributes dataset
which can attach to the packet table?

Is it that I am just to bound by my conventional view of the world, and
HDF is more general. Can I make the packetTable actually anything
I want it to be via attributes?

Thanks,
Michele

--
Michele D. De La Pena Scientific Software Engineer
Large Binocular Telescope Obs. email: mdelapena@lbto.org
933 N. Cherry Avenue phone: 520-626-1402
Tucson, AZ 85721-0065 cell: 520-419-8825

------------------------------

Email addresses of ITT Exelis employees have changed from itt.com to
exelisinc.com. Please update your favorites and contact information to
reflect these changes.

This e-mail and any files transmitted with it may be proprietary and are
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this e-mail in error please notify the
sender. Please note that any views or opinions presented in this e-mail are
solely those of the author and do not necessarily represent those of Exelis
Inc. The recipient should check this e-mail and any attachments for the
presence of viruses. Exelis Inc. accepts no liability for any damage caused
by any virus transmitted by this e-mail.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Michele D. De La Pena Scientific Software Engineer
Large Binocular Telescope Obs. email: mdelapena@lbto.org
933 N. Cherry Avenue phone: 520-626-1402
Tucson, AZ 85721-0065 cell: 520-419-8825

There is no need to actually close the packet table (H5PT) version of the dataset before opening as an H5D. You just need to open it like it was an existing dataset, file/groupID and name. HDF Group folks can correct me, but I think it is the accepted method for the moment. It would be nice if it worked directly on the H5PT ID as the documentation says (or said) it does, but last I heard there wasn't funding & enough interest to make the (rather considerable) change to the underlying code.

FWIW, you also need to use this trick if you attach dimension scales to the packet table.

Scott

···

From: Hdf-forum [mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Michele De La Pena
Sent: Tuesday, October 23, 2012 5:13 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Newbie questions regarding tables

HI Again,

I guess I need a pointer from you. I have created by first HDF5 file, and it contains a packet table (yeah).
When I tried to attach attributes, I got an error

HDF5-DIAG: Error detected in HDF5 (1.9.131) thread 0:
  #000: H5A.c line 242 in H5Acreate2(): not a location
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: H5Gloc.c line 253 in H5G_loc(): invalid object ID
    major: Invalid arguments to routine
    minor: Bad value

From my searching on the Web and your kind warning below (I fell into the hole anyway), I see

what you mean. However, I am not exactly sure of your solution. Do you mean while I created
the packet table as

hid_t ptTable = H5PTcreate_fl (fileID, "PCS Weather Table", memType, chunkSize, compress);

that I need to close the table, and then open the table as a dataset as below?

hid_t ptAsDs = H5Dopen (ptTable, "PCS Weather Table", H5P_DEFAULT);

Use of ptTable does not work in this open statement. I tried using the file identifier in the
H5Dopen in conjunction with the packet table name, and this seems to work (I have other
errors). Is this the accepted way to do this?

Thanks,
Michele

On Tue, Oct 23, 2012 at 7:41 AM, Mitchell, Scott - IS <Scott.Mitchell@exelisinc.com<mailto:Scott.Mitchell@exelisinc.com>> wrote:
Michele,

I'm dealing with a similar timestamped data streams and use Packet Tables almost exclusively.

My solution for the unit labels problem was to attach an attribute to the Packet Table. The Unit attribute is an array of strings, one per column.

Another important trick, for HDF5.net at least, you need to open the Packet Table as a (H5D) dataset to attach/read an attribute.

I'm not sure what you mean by (4) Packet Tables not having the concept of a column?

Scott

From: Hdf-forum [mailto:hdf-forum-bounces@hdfgroup.org<mailto:hdf-forum-bounces@hdfgroup.org>] On Behalf Of Michele De La Pena
Sent: Monday, October 22, 2012 5:54 PM
To: hdf-forum@hdfgroup.org<mailto:hdf-forum@hdfgroup.org>
Subject: [Hdf-forum] Newbie questions regarding tables

HI Folks,

I am an HDF5 newbie, so please be patient. I have a few basic
questions after reading the tutorials and such. First, I will say that
I have telemetry data (timestamp with state information) which I need
to stream to an HDF file. The data nicely fits the model of a table.

My Sys Admin has installed HDF5 version 1.9.131.

My first question is regard to your table APIs. I see that an HDF
file can consist of the fundamental constructs of group and/or dataset.
However, it seems the table and packetTable are also fundamental in
that I do not have to package either of these tables inside of a group to
make a valid HDF file. (1) Is this the way these are intended to be used?

My problem with the regular table is that during its creation, it does
not hand back an identifier so I cannot attach attributes directly to
the table. I would like the columns in the table to have "units" and
a "description" as meta-data for the table. I can put the regular
table and an attributes into a group (where the attributes
are attached to the group) which seems not too bad overall.
(2) Is this what one would ordinarily do for this issue?
(3) Maybe I made a mistake, but I cannot h5dump or h5stat the file I
have generated with a regular table. Is this expected?

(4) As for the packet table, I can attach attributes, but the packet table
does not really have the concept of columns as far as I can tell. So
is it assumed that meta-data will be handled in the attributes dataset
which can attach to the packet table?

Is it that I am just to bound by my conventional view of the world, and
HDF is more general. Can I make the packetTable actually anything
I want it to be via attributes?

Thanks,
Michele

--
Michele D. De La Pena Scientific Software Engineer
Large Binocular Telescope Obs. email: mdelapena@lbto.org<mailto:mdelapena@lbto.org>
933 N. Cherry Avenue phone: 520-626-1402<tel:520-626-1402>
Tucson, AZ 85721-0065 cell: 520-419-8825<tel:520-419-8825>

________________________________

Email addresses of ITT Exelis employees have changed from itt.com<http://itt.com> to exelisinc.com<http://exelisinc.com>. Please update your favorites and contact information to reflect these changes.

This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Exelis Inc. The recipient should check this e-mail and any attachments for the presence of viruses. Exelis Inc. accepts no liability for any damage caused by any virus transmitted by this e-mail.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org<mailto:Hdf-forum@hdfgroup.org>
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Michele D. De La Pena Scientific Software Engineer
Large Binocular Telescope Obs. email: mdelapena@lbto.org<mailto:mdelapena@lbto.org>
933 N. Cherry Avenue phone: 520-626-1402<tel:520-626-1402>
Tucson, AZ 85721-0065 cell: 520-419-8825<tel:520-419-8825>