Re: Apple atan2f

[Michael D Godfrey]

On 11/10/2010 08:27 PM, Rik wrote:

Imagine representing pi in base10 with only four decimal digits.
pi = 3.14159265...
pi(nearest value) = 3.1416
pi(round->0) = 3.1415

Interestingly, the 754 specification does allow 4 other types of rounding
rules, including rounding towards zero.  But, if Apple wants to use
rounding towards zero it needs to do that uniformly across all their math
functions.

Here are results from Octave on a Linux machine:
octave:1> almostpi = +3.1415924;
octave:2> format hex
octave:3> single(almostpi)
ans = 400921fb40000000
octave:4> single(pi)
ans = 400921fb60000000
octave:5> pi
ans = 400921fb54442d18

The value of pi from double precision shows that it needs a hex '5' in the
last position.  Unfortunately, the single precision does not have the ones
bit so the final hex digit must be either 4 or 6.  Linux seems to follow
the IEEE specification and rounds to the nearest value which is 6.

--Rik

It is worth asking: for what value of n does pi - n*eps become different from pi
using round to zero against round to nearest, or other choices.  Obviously in
this case n for round to zero is a lot bigger.  Iterative algorithms which depend
on differences need to consider when they are within a loss of significance
due to rounding region.  I am really surprised that Intel has made this choice.

Michael