Any road map towards a portable TIME type in HDF5?

Steve_Bissell · May 26, 2016, 1:16pm

Is there any roadmap within HDF5 to implement / define a portable time type? H5T_TIME is clearly not portable, and from python forum reports I see that python, at least, regards HDF5 as lacking a basic time type.
Alternatively, is there a wide-spread "de facto" that most people are using to represent time - e.g. double?

To be clear, I'm referring to a no-frills "seconds since start of epoch" type of time with resolution down to (at least) microseconds, which would be used for "timestamping" data. Given the user base that HDF5 supports, this problem must have come up many times, with more demanding sub-second resolution.

Currently, we use a composite, but this means that the time field always has to be treated in a "special" way, which is unsatisfactory. And at the most trivial level, HDFView allows a quick sanity check of data by plotting one column against another as abscissa .... except in the case of our composite time, where I'm limited to plotting against the seconds part of the structure.

<html><head></head><body><font color="black" face="arial" size="2">
This email and its attachments may contain confidential and/or privileged information. If you have received them in error you must not use, copy or disclose their content to any person. Please notify the sender immediately and then delete this email from your system. This e-mail has been scanned for viruses, but it is the responsibility of the recipient to conduct their own security measures. Airbus Operations Limited is not liable for any loss or damage arising from the receipt or use of this e-mail.

Airbus Operations Limited, a company registered in England and Wales, registration number, 3468788. Registered office: Pegasus House, Aerospace Avenue, Filton, Bristol, BS34 7PA, UK.
</font>
</body>
</html>

mark.koennecke · May 26, 2016, 3:20pm

Dear Stephen Bissel,

···

Am 26.05.2016 um 15:16 schrieb BISSELL, Stephen (AEROCONSEIL SA) <stephen.bissell.external@airbus.com<mailto:stephen.bissell.external@airbus.com>>:

Is there any roadmap within HDF5 to implement / define a portable time type? H5T_TIME is clearly not portable, and from python forum reports I see that python, at least, regards HDF5 as lacking a basic time type.
Alternatively, is there a wide-spread “de facto” that most people are using to represent time – e.g. double?

To be clear, I’m referring to a no-frills “seconds since start of epoch” type of time with resolution down to (at least) microseconds, which would be used for “timestamping” data. Given the user base that HDF5 supports, this problem must have come up many times, with more demanding sub-second resolution.

Currently, we use a composite, but this means that the time field always has to be treated in a “special” way, which is unsatisfactory. And at the most trivial level, HDFView allows a quick sanity check of data by plotting one column against another as abscissa …. except in the case of our composite time, where I’m limited to plotting against the seconds part of the structure.

This email and its attachments may contain confidential and/or privileged information. If you have received them in error you must not use, copy or disclose their content to any person. Please notify the sender immediately and then delete this email from your system. This e-mail has been scanned for viruses, but it is the responsibility of the recipient to conduct their own security measures. Airbus Operations Limited is not liable for any loss or damage arising from the receipt or use of this e-mail.

Airbus Operations Limited, a company registered in England and Wales, registration number, 3468788. Registered office: Pegasus House, Aerospace Avenue, Filton, Bristol, BS34 7PA, UK.

we use a double calculated with the function given below:

double DoubleTime(void)
{
  struct timeval now;
  /* the resolution of this function is usec, if the machine supports this
     and the mantissa of a double is 51 bits or more (31 bits for seconds
     and 20 for microseconds)
   */
  gettimeofday(&now, NULL);
  return now.tv_sec + now.tv_usec / 1e6;
}

The code is for unix, of course, on windows your mileage will vary.

Best Regards,

Mark Koennecke

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

paramon · May 30, 2016, 3:57pm

26.05.2016 16:16, BISSELL, Stephen (AEROCONSEIL SA) пишет:

Is there any roadmap within HDF5 to implement / define a portable time
type? H5T_TIME is clearly not portable, and from python forum reports I
see that python, at least, regards HDF5 as lacking a basic time type.
Alternatively, is there a wide-spread “de facto” that most people are
using to represent time – e.g. double?

To be clear, I’m referring to a no-frills “seconds since start of epoch”
type of time with resolution down to (at least) microseconds, which
would be used for “timestamping” data. Given the user base that HDF5
supports, this problem must have come up many times, with more demanding
sub-second resolution.

Currently, we use a composite, but this means that the time field always
has to be treated in a “special” way, which is unsatisfactory. And at
the most trivial level, HDFView allows a quick sanity check of data by
plotting one column against another as abscissa …. except in the case of
our composite time, where I’m limited to plotting against the seconds
part of the structure.

Hello Stephen!

It seems that perfect universal date/time format doesn't exist. Based on application, ISO8601 or Unix time might be most appropriate. A good example of how date/time might be handled is SQLite:

https://www.sqlite.org/datatype3.html#section_2_2

My personal experience suggests that ISO8601 is best (but only if performance is not critical).

Best wishes,
Andrey Paramonov

···

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

David1 · May 26, 2016, 3:35pm

In my humble opinion time since epoch is not a good standard. Following is
the best practice I could come up with for our time needs. We store times
in ISO 8601 as UTC. You can use a decimal in the seconds field if for high
resolution. Since we want to know local time where the data was captured we
also store timezone info next to it but not instead of UTC. UTC is always
primary. For data sets of time info we use a double with seconds since the
ISO stamp. This is working well for us.

David

···

On Thu, May 26, 2016 at 8:20 AM, Koennecke Mark (PSI) <mark.koennecke@psi.ch > wrote:

Dear Stephen Bissel,

Am 26.05.2016 um 15:16 schrieb BISSELL, Stephen (AEROCONSEIL SA) < > stephen.bissell.external@airbus.com>:

Is there any roadmap within HDF5 to implement / define a portable time
type? H5T_TIME is clearly not portable, and from python forum reports I see
that python, at least, regards HDF5 as lacking a basic time type.
Alternatively, is there a wide-spread “de facto” that most people are
using to represent time – e.g. double?

To be clear, I’m referring to a no-frills “seconds since start of epoch”
type of time with resolution down to (at least) microseconds, which would
be used for “timestamping” data. Given the user base that HDF5 supports,
this problem must have come up many times, with more demanding sub-second
resolution.

Currently, we use a composite, but this means that the time field always
has to be treated in a “special” way, which is unsatisfactory. And at the
most trivial level, HDFView allows a quick sanity check of data by plotting
one column against another as abscissa …. except in the case of our
composite time, where I’m limited to plotting against the seconds part of
the structure.

This email and its attachments may contain confidential and/or privileged information. If you have received them in error you must not use, copy or disclose their content to any person. Please notify the sender immediately and then delete this email from your system. This e-mail has been scanned for viruses, but it is the responsibility of the recipient to conduct their own security measures. Airbus Operations Limited is not liable for any loss or damage arising from the receipt or use of this e-mail.

Airbus Operations Limited, a company registered in England and Wales, registration number, 3468788. Registered office: Pegasus House, Aerospace Avenue, Filton, Bristol, BS34 7PA, UK.

we use a double calculated with the function given below:

double DoubleTime(void)
{
  struct timeval now;
  /* the resolution of this function is usec, if the machine supports this

     and the mantissa of a double is 51 bits or more (31 bits for seconds

     and 20 for microseconds)

   */
  gettimeofday(&now, NULL);
  return now.tv_sec + now.tv_usec / 1e6;
}

The code is for unix, of course, on windows your mileage will vary.

Best Regards,

     Mark Koennecke

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Steve_Bissell · May 27, 2016, 8:20am

@David
Indeed, I came to that conclusion myself whilst pondering the limitations of double. Our current implementation was too close a copy of a previous, legacy format. Better would be to store the start time as absolute (UTC in ISO 8601 perhaps) and then mark up the individual packets as double seconds-since-start (which in our case, could be negative). Thanks for the input.

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of David
Sent: 26 May 2016 16:36
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Any road map towards a portable TIME type in HDF5?

In my humble opinion time since epoch is not a good standard. Following is the best practice I could come up with for our time needs. We store times in ISO 8601 as UTC. You can use a decimal in the seconds field if for high resolution. Since we want to know local time where the data was captured we also store timezone info next to it but not instead of UTC. UTC is always primary. For data sets of time info we use a double with seconds since the ISO stamp. This is working well for us.
David

On Thu, May 26, 2016 at 8:20 AM, Koennecke Mark (PSI) <mark.koennecke@psi.ch<mailto:mark.koennecke@psi.ch>> wrote:
Dear Stephen Bissel,

Am 26.05.2016 um 15:16 schrieb BISSELL, Stephen (AEROCONSEIL SA) <stephen.bissell.external@airbus.com<mailto:stephen.bissell.external@airbus.com>>:

Is there any roadmap within HDF5 to implement / define a portable time type? H5T_TIME is clearly not portable, and from python forum reports I see that python, at least, regards HDF5 as lacking a basic time type.
Alternatively, is there a wide-spread “de facto” that most people are using to represent time – e.g. double?

To be clear, I’m referring to a no-frills “seconds since start of epoch” type of time with resolution down to (at least) microseconds, which would be used for “timestamping” data. Given the user base that HDF5 supports, this problem must have come up many times, with more demanding sub-second resolution.

Currently, we use a composite, but this means that the time field always has to be treated in a “special” way, which is unsatisfactory. And at the most trivial level, HDFView allows a quick sanity check of data by plotting one column against another as abscissa …. except in the case of our composite time, where I’m limited to plotting against the seconds part of the structure.

This email and its attachments may contain confidential and/or privileged information. If you have received them in error you must not use, copy or disclose their content to any person. Please notify the sender immediately and then delete this email from your system. This e-mail has been scanned for viruses, but it is the responsibility of the recipient to conduct their own security measures. Airbus Operations Limited is not liable for any loss or damage arising from the receipt or use of this e-mail.

Airbus Operations Limited, a company registered in England and Wales, registration number, 3468788. Registered office: Pegasus House, Aerospace Avenue, Filton, Bristol, BS34 7PA, UK.

we use a double calculated with the function given below:

double DoubleTime(void)
{
  struct timeval now;
  /* the resolution of this function is usec, if the machine supports this
     and the mantissa of a double is 51 bits or more (31 bits for seconds
     and 20 for microseconds)
   */
  gettimeofday(&now, NULL);
  return now.tv_sec + now.tv_usec / 1e6;
}

The code is for unix, of course, on windows your mileage will vary.

Best Regards,

Mark Koennecke

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

<html><head></head><body><font color="black" face="arial" size="2">
This email and its attachments may contain confidential and/or privileged information. If you have received them in error you must not use, copy or disclose their content to any person. Please notify the sender immediately and then delete this email from your system. This e-mail has been scanned for viruses, but it is the responsibility of the recipient to conduct their own security measures. Airbus Operations Limited is not liable for any loss or damage arising from the receipt or use of this e-mail.

Airbus Operations Limited, a company registered in England and Wales, registration number, 3468788. Registered office: Pegasus House, Aerospace Avenue, Filton, Bristol, BS34 7PA, UK.
</font>
</body>
</html>

sean · May 27, 2016, 6:56pm

It all depends what you want to do and why, and that solution is obviously good in many cases. But remember that calendar dates like that are not always increasing, or even unique. ex: in much of North America when clocks go back in the autumn, there are two 01:00 in a day that has 25 hours. So in many use cases, a (micro)second counter from epoch is preferable if you're going to be doing elapsed time calculations for example.

Cheers,

···

On Thu, 26 May 2016 08:35:35 -0700, David said:

In my humble opinion time since epoch is not a good standard. Following is
the best practice I could come up with for our time needs. We store times
in ISO 8601 as UTC. You can use a decimal in the seconds field if for high
resolution. Since we want to know local time where the data was captured we
also store timezone info next to it but not instead of UTC. UTC is always
primary. For data sets of time info we use a double with seconds since the
ISO stamp. This is working well for us.

--
____________________________________________________________
Sean McBride, B. Eng sean@rogue-research.com
Rogue Research www.rogue-research.com
Mac Software Developer Montréal, Québec, Canada

Scott_Mitchell · May 31, 2016, 12:55pm

For our Windows (.NET) case I’m storing the .NET DateTime.Ticks. It’s similar but different from the Epoch time (it’s a long counting the number of 100 ns since midnight 1/1/0001).We aren’t storing time zone info, everything is in UTC. And the users requested a date time string, so that’s recorded alongside. Though technically the Ticks value is official, I think the string gets used more often.

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Koennecke Mark (PSI)
Sent: Thursday, May 26, 2016 11:21 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Any road map towards a portable TIME type in HDF5?

Dear Stephen Bissel,

Am 26.05.2016 um 15:16 schrieb BISSELL, Stephen (AEROCONSEIL SA) <stephen.bissell.external@airbus.com<mailto:stephen.bissell.external@airbus.com>>:

Is there any roadmap within HDF5 to implement / define a portable time type? H5T_TIME is clearly not portable, and from python forum reports I see that python, at least, regards HDF5 as lacking a basic time type.
Alternatively, is there a wide-spread “de facto” that most people are using to represent time – e.g. double?

To be clear, I’m referring to a no-frills “seconds since start of epoch” type of time with resolution down to (at least) microseconds, which would be used for “timestamping” data. Given the user base that HDF5 supports, this problem must have come up many times, with more demanding sub-second resolution.

Currently, we use a composite, but this means that the time field always has to be treated in a “special” way, which is unsatisfactory. And at the most trivial level, HDFView allows a quick sanity check of data by plotting one column against another as abscissa …. except in the case of our composite time, where I’m limited to plotting against the seconds part of the structure.

This email and its attachments may contain confidential and/or privileged information. If you have received them in error you must not use, copy or disclose their content to any person. Please notify the sender immediately and then delete this email from your system. This e-mail has been scanned for viruses, but it is the responsibility of the recipient to conduct their own security measures. Airbus Operations Limited is not liable for any loss or damage arising from the receipt or use of this e-mail.

Airbus Operations Limited, a company registered in England and Wales, registration number, 3468788. Registered office: Pegasus House, Aerospace Avenue, Filton, Bristol, BS34 7PA, UK.

we use a double calculated with the function given below:

double DoubleTime(void)
{
  struct timeval now;
  /* the resolution of this function is usec, if the machine supports this
     and the mantissa of a double is 51 bits or more (31 bits for seconds
     and 20 for microseconds)
   */
  gettimeofday(&now, NULL);
  return now.tv_sec + now.tv_usec / 1e6;
}

The code is for unix, of course, on windows your mileage will vary.

Best Regards,

Mark Koennecke

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5