[Octave-bug-tracker] [bug #46262] [octave forge] (communications) de2bi: patch for bizarre results with non-float input

[anonymous]

Follow-up Comment #2, bug #46262 (project octave):

The root cause of this problem is in the line

  b = floor (rem (d, p*power) ./ power);

When d is an integer type the result is rounded before the floor operation can
truncate it.  The patch attached with the original bug report addressed this
problem by forcing the divide to be done as floating point, so that rounding
does not occur before the floor operation.

That is fine for integer types with fewer than 54 bits.  Larger integers
(i.e., uint64) that use more than 54 bits cannot be represented exactly in a
double precision floating point number, so the answer will be incorrect.

Changing the above line to:

  b = idivide(rem (d, p*power), power, 'floor')

seems to fix the problem for the limited cases I have tried.

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?46262>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/