[Octave-bug-tracker] [bug #61143] Functions sum and mean returns wrong answer for single precision input

[Michael Leitner]

Follow-up Comment #33, bug #61143 (project octave):

Yes, this is far from a patch, as I am much too little familiar with the
internals of octave to touch something like the fundamental internal sum
algorithm. It is just meant to show that you can essentially solve all
problems with summation (yes, also for pathological inputs) without losing
efficiency, so that if somebody comes around with a) the necessary knowledge
of the octave internals and b) the motivation to do this, then this
implementation can serve as a guide. Of course, if any user should really have
a problem where a robust and efficient summation algorithm is needed, this one
can be compiled as external oct-file and used directly.

But to go back to the very original problem (which still makes up the title,
namely the precision of summing singles): as a quick solution I propose
switching the default summation algorithm of singles to "double" -- this would
be very little effort, is at least as fast, and more accurate. Note that we
never specified which algorithm we use, so we never gave a guarantee exactly
which result would be returned, thus nobody can complain about a better
accuracy. I think this would save some hours of head-scratching on the user
side.

By the way: mean has an optional argument outtype. According to the
documentation, this determines the precision of the output, but does it also
determine the precision of the summation algorithm? If yes, I would suggest to
change the documentation, if not, I would suggest to either do the summation
in the output precision, or better do it always at least in double. The
original report seems to indicate that even when requesting double output for
mean, the summation is still done in single.


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?61143>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/