[Octave-bug-tracker] [bug #54567] median(uint8) gives bad results

[Dan Sebald]

Follow-up Comment #7, bug #54567 (project octave):

Nice article on mean computation...  This is why I wondered if it were better
to put this case of 64-bit average in the mean() script and call mean() inside
median() instead.  Could there be an internal library routine that is stable
that making a built-in mean() makes sense?

Still, here for median() it is only two numbers we are averaging, plus the
formula in the reference pertains to float values, not integer "wrap" effects.
 Consider that the formula from the reference becomes


mu_2 = mu_1 + 1/2 (x_2 - mu_1)
     = x_1  + 1/2 (x_2 - x_1)


which is the same formula as in Comment #4.

KT, keep thinking about the formulas; it shouldn't be too difficult to
implement conditionals with scripts.  They slow things down, but I'd prefer
being correct than have a warning message.  For example, the formula of
Comment #4 could be implemented as

retval = min (x, dim) + (max (x, dim) - min (x, dim))/2;

Maybe there is a way to generalize the formula from the reference for
integers.  The issue with integers is the loss of the fractional portion when
doing the intermediate mean in the recursion.  Perhaps there is a way to
accumulate those lost bits as some sort of residual and then at the very end
of the computation adjust the final computation with the residual divided by
N.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54567>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/