The documentation at
http://www.hdfgroup.org/HDF5/doc/UG/10_Datasets.html#Scaleoffset
states:
"The relative error for each value should be no more than 5*
(10^(D-scaling factor +1))."
Based on the description of the D-scaling algorithm, it seems the
above statement is incorrect (scale factor of 2 produces relative
error of 5000?). If I have an array of floats "a", the round trip
produces an error of:
a-(round(a-min(a))*10^scalefactor)/10^scalefactor+min(a))
Testing yields an *absolute* error (not relative error) of
5*10^-(scalefactor+1) (note the minus sign).
Could someone please comment? This question is related to a
contribution to h5py, we just want to get the documentation right
before releasing it (see https://github.com/h5py/h5py/pull/255).
Thank you,
Darren