Half precision floating-point format

Dimitris_Servis · October 23, 2009, 5:05pm

Hi all

I intend to use the Half precision floating-point format for storing data
that do not need as much precision in order to save space. Anyone has
experience with this and HDF5? From what I have seen it is not supported.
Any plans to support it or should I implement a user defined atomic type?
Thanks a lot!

Regards,

-- dimitris

werner · October 23, 2009, 5:09pm

This question came up before, it seems the half precision floating point is
already supported in HDF5, just not documented (or, not in the list of
H5T_... types). I have not tried it yet, but all the conversions and such
are said to work already.

There might even be some way to generically define floating point types
of arbitrary precision, but I might be wrong in that. Support for quad
and oct precision types could also be cute

Werner

···

On Fri, 23 Oct 2009 12:05:03 -0500, Dimitris Servis <servisster@gmail.com> wrote:

Hi all

I intend to use the Half precision floating-point format for storing data
that do not need as much precision in order to save space. Anyone has
experience with this and HDF5? From what I have seen it is not supported.
Any plans to support it or should I implement a user defined atomic type?
Thanks a lot!

Regards,

-- dimitris

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

yue.nicholas · October 29, 2009, 2:24am

Dimitris Servis wrote:

Hi all

I intend to use the Half precision floating-point format for storing data that do not need as much precision in order to save space. Anyone has experience with this and HDF5? From what I have seen it is not supported. Any plans to support it or should I implement a user defined atomic type? Thanks a lot!

Hi Dimitris,

I am also keen for HDF5 to support half-float as we use OpenEXR and Imath for a lot of the work we are doing.

Half precision, is in the IEEE 754-2008 standard <http://en.wikipedia.org/wiki/IEEE_754-2008> so I am hoping it will be supported in HDF5

Please post a summary when you have gotten your feedback.

Thanks.

Regards

···

Regards,

-- dimitris
------------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

werner · October 29, 2009, 2:31am

It appears the HDF5 API allows for an even wider range of floating point
formats than just half-float, as all those properties can be defined via
the H5T interface on atomic data types:

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-SetPrecision
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-SetEbias

etc.

Just there is no H5T_NATIVE_HALF pre-defined datatype, though it would
certainly be possible to define a user-defined atomic datatype which
is equivalent to the half-float in OpenEXR.

Werner

···

On Wed, 28 Oct 2009 21:24:48 -0500, Nicholas Yue <yue.nicholas@gmail.com> wrote:

Dimitris Servis wrote:

Hi all

I intend to use the Half precision floating-point format for storing
data that do not need as much precision in order to save space. Anyone
has experience with this and HDF5? From what I have seen it is not
supported. Any plans to support it or should I implement a user
defined atomic type? Thanks a lot!

Hi Dimitris,

    I am also keen for HDF5 to support half-float as we use OpenEXR and
Imath for a lot of the work we are doing.

    Half precision, is in the IEEE 754-2008 standard
<http://en.wikipedia.org/wiki/IEEE_754-2008> so I am hoping it will be
supported in HDF5

    Please post a summary when you have gotten your feedback.

    Thanks.

Regards

Regards,

-- dimitris
------------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

Dimitris_Servis · October 29, 2009, 4:37pm

Hi all,

I haven't replied (nor thanked Werner) to this yet as I am still looking
into it and it's one issue on my pile (I guess it's the same for everyone,
isn't it ) Sure, I know I can define a user-defined type, my initial
question was exactly this, if I had to do it on my own or if it is/will be
supported from the library (like IEEE 32 and 64 floats). There may be some
room for optimization if it is done on the part of the library as
half-precision floating point numbers are not supported by the language so
you'd have to just load them in a buffer or convert them to floats or
doubles. another issue is that the largest half precision float is 65504
which means that some scaling must be done to fit any float or double value
range to this. Anyway, I'll keep you posted on what I plan to do, but any
ideas are always welcome.

Best Regards

-- dimitris

···

2009/10/29 Werner Benger <werner@cct.lsu.edu>

It appears the HDF5 API allows for an even wider range of floating point
formats than just half-float, as all those properties can be defined via
the H5T interface on atomic data types:

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-SetPrecision
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-SetEbias

etc.

Just there is no H5T_NATIVE_HALF pre-defined datatype, though it would
certainly be possible to define a user-defined atomic datatype which
is equivalent to the half-float in OpenEXR.

       Werner

On Wed, 28 Oct 2009 21:24:48 -0500, Nicholas Yue <yue.nicholas@gmail.com> > wrote:

Dimitris Servis wrote:

Hi all

I intend to use the Half precision floating-point format for storing
data that do not need as much precision in order to save space. Anyone
has experience with this and HDF5? From what I have seen it is not
supported. Any plans to support it or should I implement a user
defined atomic type? Thanks a lot!

Hi Dimitris,

   I am also keen for HDF5 to support half-float as we use OpenEXR and
Imath for a lot of the work we are doing.

   Half precision, is in the IEEE 754-2008 standard
<http://en.wikipedia.org/wiki/IEEE_754-2008> so I am hoping it will be
supported in HDF5

   Please post a summary when you have gotten your feedback.

   Thanks.

Regards

Regards,

-- dimitris
------------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

werner · October 29, 2009, 4:45pm

Hi Dimitris,

as far as I know (but didn't test it myself) the HDF5 conversion routines
can already be used with user-defined floating point types, so you can just
store a float array as halfs and vice versa, should require no more than an
initial type definition. The question is anyway, once you have a set of halfs
in RAM, what do to with them? Of course can silently pass them to the GPU,
if you don't have a half precision math library, so what HDF5 offers in terms of
storage solution might just be sufficient. With the user-defined types, you
could even go further and first determine the min/max of a float field,
set up an appropriate 16bit float, and store it in the file, while during
retrieval it will automatically be converted back into float.

Might need confirmation from the HDF5 group if that really works this way,
but I would expect so.

Though I agree, it would be nice to have the standard half floats as
used on GPU's as a predefined HDF5 type.

Werner

···

On Thu, 29 Oct 2009 11:37:42 -0500, Dimitris Servis <servisster@gmail.com> wrote:

Hi all,

I haven't replied (nor thanked Werner) to this yet as I am still looking
into it and it's one issue on my pile (I guess it's the same for everyone,
isn't it ) Sure, I know I can define a user-defined type, my initial
question was exactly this, if I had to do it on my own or if it is/will be
supported from the library (like IEEE 32 and 64 floats). There may be some
room for optimization if it is done on the part of the library as
half-precision floating point numbers are not supported by the language so
you'd have to just load them in a buffer or convert them to floats or
doubles. another issue is that the largest half precision float is 65504
which means that some scaling must be done to fit any float or double value
range to this. Anyway, I'll keep you posted on what I plan to do, but any
ideas are always welcome.

Best Regards

-- dimitris

2009/10/29 Werner Benger <werner@cct.lsu.edu>

It appears the HDF5 API allows for an even wider range of floating point
formats than just half-float, as all those properties can be defined via
the H5T interface on atomic data types:

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-SetPrecision
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-SetEbias

etc.

Just there is no H5T_NATIVE_HALF pre-defined datatype, though it would
certainly be possible to define a user-defined atomic datatype which
is equivalent to the half-float in OpenEXR.

       Werner

On Wed, 28 Oct 2009 21:24:48 -0500, Nicholas Yue <yue.nicholas@gmail.com> >> wrote:

Dimitris Servis wrote:

Hi all

I intend to use the Half precision floating-point format for storing
data that do not need as much precision in order to save space. Anyone
has experience with this and HDF5? From what I have seen it is not
supported. Any plans to support it or should I implement a user
defined atomic type? Thanks a lot!

Hi Dimitris,

   I am also keen for HDF5 to support half-float as we use OpenEXR and
Imath for a lot of the work we are doing.

   Half precision, is in the IEEE 754-2008 standard
<http://en.wikipedia.org/wiki/IEEE_754-2008> so I am hoping it will be
supported in HDF5

   Please post a summary when you have gotten your feedback.

   Thanks.

Regards

Regards,

-- dimitris
------------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

epourmal · October 29, 2009, 5:10pm

Hi Dimitris and Werner,

To take full advantage in storing user-defined types, one should use N-bit filter (H5Pset_nbit) when dataset is created (see http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetNbit).
As Werner mentioned, HDF5 library ssupport datatype conversions between user-defined types and other types. This feature is tested in the library, but of course, there is always room for improvement

And we noted the request for H5T_NATIVE_HALF (cannot promise, but will see what we can do).

Elena

···

On Oct 29, 2009, at 11:45 AM, Werner Benger wrote:

Hi Dimitris,

as far as I know (but didn't test it myself) the HDF5 conversion routines
can already be used with user-defined floating point types, so you can just
store a float array as halfs and vice versa, should require no more than an
initial type definition. The question is anyway, once you have a set of halfs
in RAM, what do to with them? Of course can silently pass them to the GPU,
if you don't have a half precision math library, so what HDF5 offers in terms of
storage solution might just be sufficient. With the user-defined types, you
could even go further and first determine the min/max of a float field,
set up an appropriate 16bit float, and store it in the file, while during
retrieval it will automatically be converted back into float.

Might need confirmation from the HDF5 group if that really works this way,
but I would expect so.

Though I agree, it would be nice to have the standard half floats as
used on GPU's as a predefined HDF5 type.

  Werner

On Thu, 29 Oct 2009 11:37:42 -0500, Dimitris Servis <servisster@gmail.com > > wrote:

Hi all,

I haven't replied (nor thanked Werner) to this yet as I am still looking
into it and it's one issue on my pile (I guess it's the same for everyone,
isn't it ) Sure, I know I can define a user-defined type, my initial
question was exactly this, if I had to do it on my own or if it is/will be
supported from the library (like IEEE 32 and 64 floats). There may be some
room for optimization if it is done on the part of the library as
half-precision floating point numbers are not supported by the language so
you'd have to just load them in a buffer or convert them to floats or
doubles. another issue is that the largest half precision float is 65504
which means that some scaling must be done to fit any float or double value
range to this. Anyway, I'll keep you posted on what I plan to do, but any
ideas are always welcome.

Best Regards

-- dimitris

2009/10/29 Werner Benger <werner@cct.lsu.edu>

It appears the HDF5 API allows for an even wider range of floating point
formats than just half-float, as all those properties can be defined via
the H5T interface on atomic data types:

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-SetPrecision
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-SetEbias

etc.

Just there is no H5T_NATIVE_HALF pre-defined datatype, though it would
certainly be possible to define a user-defined atomic datatype which
is equivalent to the half-float in OpenEXR.

      Werner

On Wed, 28 Oct 2009 21:24:48 -0500, Nicholas Yue <yue.nicholas@gmail.com >>> > >>> wrote:

Dimitris Servis wrote:

Hi all

I intend to use the Half precision floating-point format for storing
data that do not need as much precision in order to save space. Anyone
has experience with this and HDF5? From what I have seen it is not
supported. Any plans to support it or should I implement a user
defined atomic type? Thanks a lot!

Hi Dimitris,

  I am also keen for HDF5 to support half-float as we use OpenEXR and
Imath for a lot of the work we are doing.

  Half precision, is in the IEEE 754-2008 standard
<http://en.wikipedia.org/wiki/IEEE_754-2008> so I am hoping it will be
supported in HDF5

  Please post a summary when you have gotten your feedback.

  Thanks.

Regards

Regards,

-- dimitris
------------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Dimitris_Servis · October 29, 2009, 5:11pm

Hi Werner,

thanks again for your input! Your position is clear and I couldn't agree
more. As for the usage there are 2 usecases: the first one is load a field
and display it immediately, where you could load it directly to a buffer of
types understood by your favorite graphics engine and the other one, load it
into a buffer of doubles or floats. I think both are standard dataset
usecases for HDF5.

Best Regards,

-- dimitris

···

2009/10/29 Werner Benger <werner@cct.lsu.edu>

Hi Dimitris,

as far as I know (but didn't test it myself) the HDF5 conversion routines
can already be used with user-defined floating point types, so you can just
store a float array as halfs and vice versa, should require no more than an
initial type definition. The question is anyway, once you have a set of
halfs
in RAM, what do to with them? Of course can silently pass them to the GPU,
if you don't have a half precision math library, so what HDF5 offers in
terms of
storage solution might just be sufficient. With the user-defined types, you
could even go further and first determine the min/max of a float field,
set up an appropriate 16bit float, and store it in the file, while during
retrieval it will automatically be converted back into float.

Might need confirmation from the HDF5 group if that really works this way,
but I would expect so.

Though I agree, it would be nice to have the standard half floats as
used on GPU's as a predefined HDF5 type.

       Werner

On Thu, 29 Oct 2009 11:37:42 -0500, Dimitris Servis <servisster@gmail.com> > wrote:

Hi all,

I haven't replied (nor thanked Werner) to this yet as I am still looking
into it and it's one issue on my pile (I guess it's the same for everyone,
isn't it ) Sure, I know I can define a user-defined type, my initial
question was exactly this, if I had to do it on my own or if it is/will be
supported from the library (like IEEE 32 and 64 floats). There may be some
room for optimization if it is done on the part of the library as
half-precision floating point numbers are not supported by the language so
you'd have to just load them in a buffer or convert them to floats or
doubles. another issue is that the largest half precision float is 65504
which means that some scaling must be done to fit any float or double
value
range to this. Anyway, I'll keep you posted on what I plan to do, but any
ideas are always welcome.

Best Regards

-- dimitris

2009/10/29 Werner Benger <werner@cct.lsu.edu>

It appears the HDF5 API allows for an even wider range of floating point

formats than just half-float, as all those properties can be defined via
the H5T interface on atomic data types:

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-SetPrecision
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5T.html#Datatype-SetEbias

etc.

Just there is no H5T_NATIVE_HALF pre-defined datatype, though it would
certainly be possible to define a user-defined atomic datatype which
is equivalent to the half-float in OpenEXR.

      Werner

On Wed, 28 Oct 2009 21:24:48 -0500, Nicholas Yue <yue.nicholas@gmail.com >>> > >>> wrote:

Dimitris Servis wrote:

Hi all

I intend to use the Half precision floating-point format for storing
data that do not need as much precision in order to save space. Anyone
has experience with this and HDF5? From what I have seen it is not
supported. Any plans to support it or should I implement a user
defined atomic type? Thanks a lot!

Hi Dimitris,

  I am also keen for HDF5 to support half-float as we use OpenEXR and
Imath for a lot of the work we are doing.

  Half precision, is in the IEEE 754-2008 standard
<http://en.wikipedia.org/wiki/IEEE_754-2008> so I am hoping it will be
supported in HDF5

  Please post a summary when you have gotten your feedback.

  Thanks.

Regards

Regards,

-- dimitris

------------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--

___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University
(CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Daniel_Kahn1 · November 5, 2009, 2:00pm

Thanks Elena.
--dan