[Octave-bug-tracker] [bug #61788] arrays of type int16 contain wrong numbers

[John W. Eaton]

Follow-up Comment #1, bug #61788 (project octave):

This problem is happening because we are computing the elements of the range
using Matlab's rules for integer arithmetic that use saturation semantics.  So
when we compute


range_element(i) = base + i * increment


with base and increment both being int16, the "i * increment" part can
saturate.  In this case, the saturation value is 32767, so for all elements
beyond the one where saturation happens, you get "base + 32767".  For the
specific example, the result is 2767.

We need a better way of doing this calculation.  What would be the best way to
do it that will work naturally in a template function?

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?61788>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/