Re: Problem with compl() in gawk-3.1.3

[Paul Eggert]

Mike Romaniw <address@hidden> writes:

> I just ran some tests, and am still getting what I think are
> incorrect results.
> 
> If the input fits in 53 bits or less, everything seems fine.  If you give
> it a larger input (full 64 bits), we can see that the 11 high order bits
> have been stripped, but the low 2 bytes are still wrong.
> 
> compl(0x8888888888888888)
> 
> returns
> 
> 177777777777ff


Thanks for testing that patch.

That problem occurs because 0x8888888888888888 is not represented
exactly in Gawk; it is rounded to 0x8888888888888800.  You can verify
this by executing:

printf "%x\n", 0x8888888888888888

You will get "8888888888888800".  Hence compl is actually getting
0x8888888888888800 as its input, which explains the answer you're
seeing.  This problem is therefore independent of the bitwise
operators.

It is also somewhat independent of the hexadecimal notation.  For example,

printf "%d\n", 9838263505978427528

outputs "9838263505978427392", which is the same rounding error in decimal.

If you limit yourself to 53-bit integers or less, you'll avoid these
rounding problems (on IEEE 754 hosts, anyway).