Hello,
I would like to store in a HDF dataset all values sent by a sensor which
emits a value at every same period of time (10 ms for example).
Sometimes the sensor does not send a value and we would like to see this
gap in our HDF dataset.
Is it possible to store empty or missing values in a dataset?
We do not want to choose a special value for that purpose as we don't
know what is the range of acceptable values sent by the sensor.
Thanks for your help.
Best Regards,
Vincent Herrero
Herrero,
I think conceptually what you're referring to needing is to somehow
represent and insert NULL values into your HDF file, if you really
don't know the native valid range of the data. A true NULL represents
something very distinct from any known, valid value, akin to "no
information is available".
While routine in most relational database implementations, this
(may) not be possible using HDF's native facilities.
The default internal missing value for a HDF5 dataset is 0, but
you can currently override this exactly once only, at dataset creation
time, with a user supplied value that fits within the range of the
datasets datatype (its now a DatasetCreationProperty element).
Perhaps you can define such a value that is very clearly outside
the valid range of your sensor, as informed by its hardware design.
For example, many sensors have built in numerical limits governing
what they can emit, due to how their A/D conversion works, or to their
digital hardware/firmware if its is a pure digital sensor (e.g. a 12
or 14 bit sensor has a very deterministic limit to what it can emit),
so you could define your "pseudo-null" value as just outside this
range etc.
Ideally, the HDF facility proper could someday allow users to
specify a true NULL value as a missing value, but this can be tricky
and involve opaque conversions, since internally the value is usually
represented as some "known,valid" numeric type. HTH
Joe
On Mon, Oct 25, 2010 at 2:34 AM, Herrero Vincent <vherrero@sopragroup.com> wrote:
Hello,
I would like to store in a HDF dataset all values sent by a sensor which
emits a value at every same period of time (10 ms for example).
Sometimes the sensor does not send a value and we would like to see this gap
in our HDF dataset.Is it possible to store empty or missing values in a dataset?
We do not want to choose a special value for that purpose as we don't know
what is the range of acceptable values sent by the sensor.Thanks for your help.
Best Regards,Vincent Herrero
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
------------------------------------------------------------
Joseph Glassy
Lead Software Engineer (contractor)
NASA Measures (Freeze/Thaw),Rm CFC 424
College of Forestry and Conservation
Univ. Montana, Missoula, MT 59812
Tel: 406-243-6318 Cellular: 406-544-3315
Lupine Logic Inc.
www.lupinelogic.com
Scientific and Technical Programming
Hi Joe,
Herrero,
I think conceptually what you're referring to needing is to somehow
represent and insert NULL values into your HDF file, if you really
don't know the native valid range of the data. A true NULL represents
something very distinct from any known, valid value, akin to "no
information is available".
While routine in most relational database implementations, this
(may) not be possible using HDF's native facilities.
The default internal missing value for a HDF5 dataset is 0, but
you can currently override this exactly once only, at dataset creation
time, with a user supplied value that fits within the range of the
datasets datatype (its now a DatasetCreationProperty element).
Perhaps you can define such a value that is very clearly outside
the valid range of your sensor, as informed by its hardware design.
For example, many sensors have built in numerical limits governing
what they can emit, due to how their A/D conversion works, or to their
digital hardware/firmware if its is a pure digital sensor (e.g. a 12
or 14 bit sensor has a very deterministic limit to what it can emit),
so you could define your "pseudo-null" value as just outside this
range etc.
Ideally, the HDF facility proper could someday allow users to
specify a true NULL value as a missing value, but this can be tricky
and involve opaque conversions, since internally the value is usually
represented as some "known,valid" numeric type. HTH
Good suggestions, thanks! I just wanted to add that possibly Vincent could use the "fill value" facility in HDF5, with H5Pset_fill_value(), although I suspect that may not fulfill his requirements...
Quincey
On Oct 25, 2010, at 8:53 AM, J Glassy wrote:
Joe
On Mon, Oct 25, 2010 at 2:34 AM, Herrero Vincent > <vherrero@sopragroup.com> wrote:
Hello,
I would like to store in a HDF dataset all values sent by a sensor which
emits a value at every same period of time (10 ms for example).
Sometimes the sensor does not send a value and we would like to see this gap
in our HDF dataset.Is it possible to store empty or missing values in a dataset?
We do not want to choose a special value for that purpose as we don't know
what is the range of acceptable values sent by the sensor.Thanks for your help.
Best Regards,Vincent Herrero
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org--
------------------------------------------------------------
Joseph Glassy
Lead Software Engineer (contractor)
NASA Measures (Freeze/Thaw),Rm CFC 424
College of Forestry and Conservation
Univ. Montana, Missoula, MT 59812
Tel: 406-243-6318 Cellular: 406-544-3315Lupine Logic Inc.
www.lupinelogic.com
Scientific and Technical Programming_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
