just wanted to report that the half precision (16bit) floating point works
great with HDF5. I implemented some in-place array conversion functions (for
a good implementation with ample room of improvement see http://www.mathworks.com/matlabcentral/fileexchange/23173-ieee-754r-half-precision-floating-point-converter).
The in-place replacement is straight forward to implement. I also
implemented a half float type to use with C++ and standard containers. On
the HDF5 side, I used the in-place array converters to register the new type
using code like this:
hid_t halftype = H5Tcopy(H5T_NATIVE_FLOAT);
H5Tset_fields(halftype , 15, 10, 5, 0, 10);
H5Tset_size(halftype , 2);
H5Tregister(H5T_PERS_HARD, "half-to-float", halftype , H5T_NATIVE_FLOAT,
half_conversion_function);
(the same for all possible conversions)
So far it works great. I can convert from any array to half float and back.
Disk usage is obviously halved, so is memory usage, especially if you're on
the C++ side where it is easier to work with the custom half type.
my next attempt with the half precision and HDF5 is to make it possible to
convert data from and to half precision but also keeping a scale factor.
Data that can normally be represented with half precision can be up to 65504
(the maximum representable value). If the original data is larger than that,
it would make sense to keep a scale factor or at least the order of
magnitude before scaling. Any ideas on optimal ways to do that with HDF5?
Should I just extract a reasonable order of magnitude and keep it as an
attribute? I looked a bit at the scaling filters but they seem to suffer by
exactly the same problems!
just wanted to report that the half precision (16bit) floating point works
great with HDF5. I implemented some in-place array conversion functions (for
a good implementation with ample room of improvement see
hid_t halftype = H5Tcopy(H5T_NATIVE_FLOAT);
H5Tset_fields(halftype , 15, 10, 5, 0, 10);
H5Tset_size(halftype , 2);
H5Tregister(H5T_PERS_HARD, "half-to-float", halftype , H5T_NATIVE_FLOAT,
half_conversion_function);
(the same for all possible conversions)
So far it works great. I can convert from any array to half float and back.
Disk usage is obviously halved, so is memory usage, especially if you're on
the C++ side where it is easier to work with the custom half type.
my next attempt with the half precision and HDF5 is to make it possible to convert data from and to half precision but also keeping a scale factor. Data that can normally be represented with half precision can be up to 65504 (the maximum representable value). If the original data is larger than that, it would make sense to keep a scale factor or at least the order of magnitude before scaling. Any ideas on optimal ways to do that with HDF5? Should I just extract a reasonable order of magnitude and keep it as an attribute? I looked a bit at the scaling filters but they seem to suffer by exactly the same problems!
Will all this half precision work become part of the standard HDF5 distribution?
In the past, I have asked about half precision (binary16 in IEEE 754-2008) road map for HDF5 and got the indication it may be worked on in future so I am not sure if this is the same work.
In the past, I have asked about half precision (binary16 in IEEE
754-2008) road map for HDF5 and got the indication it may be worked on
in future so I am not sure if this is the same work.
I have been following this discussion off and on (some people have
asked me about half-precision in Python). I tested creating a custom
type and using H5Tconvert to go back and forth... it seems to work OK.
What additional features do people find HDF5 is missing for
half-precision?
We have this request in our issues database and will try to address it as time and resources permit.
We will also be happy to consider source contribution for this feature to speed-up the development.
Elena
···
On Feb 16, 2010, at 2:18 PM, Dimitris Servis wrote:
my next attempt with the half precision and HDF5 is to make it
possible to convert data from and to half precision but also keeping a
scale factor. Data that can normally be represented with half
precision can be up to 65504 (the maximum representable value). If the
original data is larger than that, it would make sense to keep a scale
factor or at least the order of magnitude before scaling. Any ideas on
optimal ways to do that with HDF5? Should I just extract a reasonable
order of magnitude and keep it as an attribute? I looked a bit at the
scaling filters but they seem to suffer by exactly the same problems!
Will all this half precision work become part of the standard HDF5
distribution?
In the past, I have asked about half precision (binary16 in IEEE
754-2008) road map for HDF5 and got the indication it may be worked on
in future so I am not sure if this is the same work.
my next attempt with the half precision and HDF5 is to make it
possible to convert data from and to half precision but also keeping a
scale factor. Data that can normally be represented with half
precision can be up to 65504 (the maximum representable value). If the
original data is larger than that, it would make sense to keep a scale
factor or at least the order of magnitude before scaling. Any ideas on
optimal ways to do that with HDF5? Should I just extract a reasonable
order of magnitude and keep it as an attribute? I looked a bit at the
scaling filters but they seem to suffer by exactly the same problems!
Will all this half precision work become part of the standard HDF5
distribution?
In the past, I have asked about half precision (binary16 in IEEE
754-2008) road map for HDF5 and got the indication it may be worked on
in future so I am not sure if this is the same work.
I would expect this to be a minor work, as half-precision is already supported by HDF5. Might be no more than to add it as a pre-defined type, ensuring that its binary layout is compatible with the half-type float that is used in OpenEXR and on modern GPU hardware.
We'd need this in one of our near-future projects anyway, so could contribute this once it's started, but someone else might already be faster.
We have this request in our issues database and will try to address it as time and resources permit.
We will also be happy to consider source contribution for this feature to speed-up the development.
Elena
On Feb 16, 2010, at 2:18 PM, Dimitris Servis wrote:
my next attempt with the half precision and HDF5 is to make it
possible to convert data from and to half precision but also keeping a
scale factor. Data that can normally be represented with half
precision can be up to 65504 (the maximum representable value). If the
original data is larger than that, it would make sense to keep a scale
factor or at least the order of magnitude before scaling. Any ideas on
optimal ways to do that with HDF5? Should I just extract a reasonable
order of magnitude and keep it as an attribute? I looked a bit at the
scaling filters but they seem to suffer by exactly the same problems!
Will all this half precision work become part of the standard HDF5
distribution?
In the past, I have asked about half precision (binary16 in IEEE
754-2008) road map for HDF5 and got the indication it may be worked on
in future so I am not sure if this is the same work.
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
I would expect this to be a minor work, as half-precision is already supported by HDF5. Might be no more than to add it as a pre-defined type, ensuring that its binary layout is compatible with the half-type float that is used in OpenEXR and on modern GPU hardware.
Is the 'half' floating-point point keyword defined in the C99 standard? (Same with the 'quad' floating-point type)
Quincey
···
On Feb 17, 2010, at 3:47 PM, Werner Benger wrote:
We'd need this in one of our near-future projects anyway, so could contribute this once it's started, but someone else might already be faster.
We have this request in our issues database and will try to address it as time and resources permit.
We will also be happy to consider source contribution for this feature to speed-up the development.
Elena
On Feb 16, 2010, at 2:18 PM, Dimitris Servis wrote:
my next attempt with the half precision and HDF5 is to make it
possible to convert data from and to half precision but also keeping a
scale factor. Data that can normally be represented with half
precision can be up to 65504 (the maximum representable value). If the
original data is larger than that, it would make sense to keep a scale
factor or at least the order of magnitude before scaling. Any ideas on
optimal ways to do that with HDF5? Should I just extract a reasonable
order of magnitude and keep it as an attribute? I looked a bit at the
scaling filters but they seem to suffer by exactly the same problems!
Will all this half precision work become part of the standard HDF5
distribution?
In the past, I have asked about half precision (binary16 in IEEE
754-2008) road map for HDF5 and got the indication it may be worked on
in future so I am not sure if this is the same work.
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
I would expect this to be a minor work, as half-precision is already supported by HDF5. Might be no more than to add it as a pre-defined type, ensuring that its binary layout is compatible with the half-type float that is used in OpenEXR and on modern GPU hardware.
Is the 'half' floating-point point keyword defined in the C99 standard? (Same with the 'quad' floating-point type)
I'm not knowledgeable in the C99 standard (sorry, mostly C++ here ), but I doubt it's there, since only GPU's implement half floats in hardware, and you can't run C99 there (as far as I know, even though CUDA might be close).
Werner
···
On Thu, 18 Feb 2010 16:44:19 +0100, Quincey Koziol <koziol@hdfgroup.org> wrote:
On Feb 17, 2010, at 3:47 PM, Werner Benger wrote:
Quincey
We'd need this in one of our near-future projects anyway, so could contribute this once it's started, but someone else might already be faster.
We have this request in our issues database and will try to address it as time and resources permit.
We will also be happy to consider source contribution for this feature to speed-up the development.
Elena
On Feb 16, 2010, at 2:18 PM, Dimitris Servis wrote:
my next attempt with the half precision and HDF5 is to make it
possible to convert data from and to half precision but also keeping a
scale factor. Data that can normally be represented with half
precision can be up to 65504 (the maximum representable value). If the
original data is larger than that, it would make sense to keep a scale
factor or at least the order of magnitude before scaling. Any ideas on
optimal ways to do that with HDF5? Should I just extract a reasonable
order of magnitude and keep it as an attribute? I looked a bit at the
scaling filters but they seem to suffer by exactly the same problems!
Will all this half precision work become part of the standard HDF5
distribution?
In the past, I have asked about half precision (binary16 in IEEE
754-2008) road map for HDF5 and got the indication it may be worked on
in future so I am not sure if this is the same work.
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
