How to write empty or missing values in a dataset ?

Joe, Quincey,

Thanks for your answers.
I will try to see if we can get a "pseudo-null" value for each sensors
we are gathering data from.
Otherwise I can always define a dataset using a H5T_COMPOUND data type
as follow:

typedef struct s1_t
{
    unsigned char emitted; // if 0 then no-val
    float value;
} s1_t;

s1_tid = H5Tcreate(H5T_COMPOUND, sizeof(s1_t));
H5Tinsert(s1_tid, "emitted", HOFFSET(s1_t, emitted), H5T_NATIVE_UCHAR);
H5Tinsert(s1_tid, "value", HOFFSET(s1_t, value), H5T_NATIVE_FLOAT);

But this solution is more size consuming.

Best Regards,

Vincent

A Tuesday 26 October 2010 09:53:38 Herrero Vincent escrigué:

Joe, Quincey,

Thanks for your answers.
I will try to see if we can get a "pseudo-null" value for each
sensors we are gathering data from.
Otherwise I can always define a dataset using a H5T_COMPOUND data
type as follow:

typedef struct s1_t
{
    unsigned char emitted; // if 0 then no-val
    float value;
} s1_t;

s1_tid = H5Tcreate(H5T_COMPOUND, sizeof(s1_t));
H5Tinsert(s1_tid, "emitted", HOFFSET(s1_t, emitted),
H5T_NATIVE_UCHAR); H5Tinsert(s1_tid, "value", HOFFSET(s1_t, value),
H5T_NATIVE_FLOAT);

But this solution is more size consuming.

If you are worried about size consumption, using compression in
combination with the shuffle filter may significantly reduce the size of
the additional 'mask' (supposing that the mask is sparse enough).

···

--
Francesc Alted

Can you instead use a time stamp on each data value? Then when you have no data, just don't write anything.

Scott

···

From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Herrero Vincent
Sent: Tuesday, October 26, 2010 3:54 AM
To: hdf-forum@hdfgroup.org
Subject: Re: [Hdf-forum] How to write empty or missing values in a dataset ?

Joe, Quincey,
Thanks for your answers.
I will try to see if we can get a "pseudo-null" value for each sensors we are gathering data from.
Otherwise I can always define a dataset using a H5T_COMPOUND data type as follow:

typedef struct s1_t
{
    unsigned char emitted; // if 0 then no-val
    float value;
} s1_t;

s1_tid = H5Tcreate(H5T_COMPOUND, sizeof(s1_t));
H5Tinsert(s1_tid, "emitted", HOFFSET(s1_t, emitted), H5T_NATIVE_UCHAR);
H5Tinsert(s1_tid, "value", HOFFSET(s1_t, value), H5T_NATIVE_FLOAT);

But this solution is more size consuming.

Best Regards,

Vincent

________________________________
This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail.

Will IEEE NaNs work? They behave a lot like database NULLs, but indeed code
that works with them needs to be aware of how to test for them and how they
behave.

···

On Tue, Oct 26, 2010 at 7:40 AM, Mitchell, Scott - IS < Scott.Mitchell@itt.com> wrote:

Can you instead use a time stamp on each data value? Then when you have
no data, just don’t write anything.

Scott

*From:* hdf-forum-bounces@hdfgroup.org [mailto:
hdf-forum-bounces@hdfgroup.org] *On Behalf Of *Herrero Vincent
*Sent:* Tuesday, October 26, 2010 3:54 AM
*To:* hdf-forum@hdfgroup.org
*Subject:* Re: [Hdf-forum] How to write empty or missing values in a
dataset ?

Joe, Quincey,

Thanks for your answers.

I will try to see if we can get a "pseudo-null" value for each sensors we
are gathering data from.

Otherwise I can always define a dataset using a H5T_COMPOUND data type as
follow:

typedef struct s1_t
{

    unsigned char emitted; // if 0 then no-val

    float value;
} s1_t;

s1_tid = H5Tcreate(H5T_COMPOUND, sizeof(s1_t));

H5Tinsert(s1_tid, "emitted", HOFFSET(s1_t, emitted), H5T_NATIVE_UCHAR);
H5Tinsert(s1_tid, "value", HOFFSET(s1_t, value), H5T_NATIVE_FLOAT);

But this solution is more size consuming.

Best Regards,

Vincent

------------------------------
This e-mail and any files transmitted with it may be proprietary and are
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this e-mail in error please notify the
sender.
Please note that any views or opinions presented in this e-mail are solely
those of the author and do not necessarily represent those of ITT
Corporation. The recipient should check this e-mail and any attachments for
the presence of viruses. ITT accepts no liability for any damage caused by
any virus transmitted by this e-mail.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
*Sebastian Good*

Hi. This is a common issue in aerospace.
There are two solutions here, depending on how reliable your data source is.
1) If you are entirely sure that your datasource emits at 10ms intervals,
then you can safely ASSUME that each record is 10ms apart. But that is an
assumption: you should justify it. Assuming that it's justified, then, as
others have suggested, there is likely, for any PHYSICAL transducer, to be a
valid range of possible outputs, and a "NULL" (aerospace equivalent
terminology: No Computed Data, or NCD) value could be chosen to be a float
outside this range. Alternatively, you could use HUGE_VAL or -HUGE_VAL (IEEE
+ or - infinity encodings) or NAN, depending on how HDF handles this.

2) If you have any doubts whatsoever about the regularity in time of your
data source, then you should store a timestamp with each record: then gaps
are obvious, as previously pointed out, by the delta in the timestamps,
which might be due to non-availability of sensor data, or irregular
scheduling of your acquisition code (assuming that you are polling the
device).

Finally, regarding file size, it is often useful to record only CHANGES (how
you define a significant change is up to you) as timestamped events. This
gives you, automatically, a very small number of records if your data
remains static over a long period, but is of limited use if your source is
"noisy" (i.e. constantly changing).

···

--
View this message in context: http://hdf-forum.184993.n3.nabble.com/How-to-write-empty-or-missing-values-in-a-dataset-tp1766106p1785322.html
Sent from the hdf-forum mailing list archive at Nabble.com.