Dear all,
Is there any convention or best practise for storing time history with
discrete values in HDF? I am recording samples from high frequency
instruments using the Packet Table API would appreciate some experience on
the best way to store the timestamps (relative).
Is it common to store timestamps with values as a compound type (as in
struct { double time; double value; })? Is it more common to store
timestamps in a separate table? How do you approach this?
Regards,
Hey karl - not sure about "best practices" but what I've done, which seems to
work pretty well so far, is indeed to use compound types, with the first
element of the struct being time. This works well for our application, where
we may be recording several thousand distinct channels that can be regarded
as asynchronous, and hence each of the packet tables storing the channel
data carries its own timestamp in the table entries.
Only thing to watch out for is that you cannot store bit-sized parameters
inside a compound type, see my post
http://hdf-forum.184993.n3.nabble.com/Possible-to-pack-bit-types-into-compound-data-types-td1131024.html#a1168717
for an example of the problem (and a very basic example of a compound type).
For our application, it's hard to see how we'd store time any other way,
since each record in the packet table arrives unpredictably, so referencing
some "other table" would just add complication storage - arrival time is an
essential property of the data itself.
Cheers
Steve
--
View this message in context: http://hdf-forum.184993.n3.nabble.com/Best-practice-for-timeseries-and-storing-timestamps-in-Packet-Table-API-tp2104069p2104301.html
Sent from the hdf-forum mailing list archive at Nabble.com.
I am doing it slightly differently. My data structure is a similar compound type depending upon the instrument data being saved. I however chose to put the time data into a second time axis (dimension scale) which is also a Packet Table. I chose this because having a smaller dataset should speed the effort to find the appropriate data index.
Initially that axis was just a timestamp, but this got pushback, and I am now also storing a date time string in the scale.
Scott
-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Steve Bissell
Sent: Friday, December 17, 2010 6:43 AM
To: hdf-forum@hdfgroup.org
Subject: Re: [Hdf-forum] Best practice for timeseries and storing timestamps
in Packet Table APIHey karl - not sure about "best practices" but what I've done, which seems to
work pretty well so far, is indeed to use compound types, with the first
element of the struct being time. This works well for our application, where
we may be recording several thousand distinct channels that can be regarded
as asynchronous, and hence each of the packet tables storing the channel
data carries its own timestamp in the table entries.
Only thing to watch out for is that you cannot store bit-sized parameters
inside a compound type, see my post
http://hdf-forum.184993.n3.nabble.com/Possible-to-pack-bit-types-into-
compound-data-types-td1131024.html#a1168717
for an example of the problem (and a very basic example of a compound type).
For our application, it's hard to see how we'd store time any other way,
since each record in the packet table arrives unpredictably, so referencing
some "other table" would just add complication storage - arrival time is an
essential property of the data itself.
Cheers
Steve
--
View this message in context: http://hdf-forum.184993.n3.nabble.com/Best-
practice-for-timeseries-and-storing-timestamps-in-Packet-Table-API-
tp2104069p2104301.html
Sent from the hdf-forum mailing list archive at Nabble.com._______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail.
Thanks Steve - I'm pleased that my original solution (compound types) was
similar to what you have used. I appreciate the heads up on bit fields -
that would probably have got me.
Cheers
On Fri, Dec 17, 2010 at 11:42 AM, Steve Bissell <stephen.bissell@airbus.com>wrote:
Hey karl - not sure about "best practices" but what I've done, which seems
to
work pretty well so far, is indeed to use compound types, with the first
element of the struct being time. This works well for our application,
where
we may be recording several thousand distinct channels that can be regarded
as asynchronous, and hence each of the packet tables storing the channel
data carries its own timestamp in the table entries.
Only thing to watch out for is that you cannot store bit-sized parameters
inside a compound type, see my posthttp://hdf-forum.184993.n3.nabble.com/Possible-to-pack-bit-types-into-compound-data-types-td1131024.html#a1168717
for an example of the problem (and a very basic example of a compound
type).
For our application, it's hard to see how we'd store time any other way,
since each record in the packet table arrives unpredictably, so referencing
some "other table" would just add complication storage - arrival time is an
essential property of the data itself.
Cheers
Steve
--
View this message in context:
http://hdf-forum.184993.n3.nabble.com/Best-practice-for-timeseries-and-storing-timestamps-in-Packet-Table-API-tp2104069p2104301.html
Sent from the hdf-forum mailing list archive at Nabble.com._______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
I had to deal with such problem, too. But with the slight difference (?)
that I had to store a large number of values for the same timestamp. And
with the knowledge, that on later reading of the data, only one/few
values related to one timestamp is/are needed.
I chose to store the timestamp and every value in dedicated datasets for
each. With a "loose" coupling by the the dataset index of the values
only. The n-th timestamp in the timestamp dataset relates to all n-th
values in the other datasets.
This reduces the i/o on reading data - as with the compound solution a
complete record is to be read even with only few of the record values
being needed.
Interesting.
Does anybody have any details of the time history API used by
Boeing[0] or the similar API supposedly used by Lockheed Martin in the
F-22 and F-35 projects[1]?
[0]: http://www.hdfgroup.org/projects/boeing/
[1]: http://goo.gl/txywD
Best wishes
On Fri, Dec 17, 2010 at 12:30 PM, Stamminger, Johannes <Johannes.Stamminger@astrium.eads.net> wrote:
I had to deal with such problem, too. But with the slight difference (?)
that I had to store a large number of values for the same timestamp. And
with the knowledge, that on later reading of the data, only one/few
values related to one timestamp is/are needed.I chose to store the timestamp and every value in dedicated datasets for
each. With a "loose" coupling by the the dataset index of the values
only. The n-th timestamp in the timestamp dataset relates to all n-th
values in the other datasets.This reduces the i/o on reading data - as with the compound solution a
complete record is to be read even with only few of the record values
being needed._______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Martin Galpin
Founder, 66laps Limited
