[Octave-bug-tracker] [bug #54567] median(uint8) gives bad results

[Dan Sebald]

Follow-up Comment #3, bug #54567 (project octave):

This is one of those I'd classify as "think about for a while".  I did notice
the "logical" case, both within median() and mean().  However, I din't think
why it is there, just that mean() with the follow-up check within median()
should still work fine.  Of course, in the alternative changeset there is an
extra call as well, to cast().

Yes, mean() is an extra function call of overhead, as mean() in turn calls
sum().  For large data this overhead should be diminished.  There's nothing
recursive or loop-like about these routines, so the greater the input matrix
size, the more CPU goes toward the inner matrix operations written in C++.

I'm fine with dropping the "native" and then use cast().  I was thinking more
along the lines of there being just one location where the sophistication of
mean() is implemented.  The one thing to be careful of, though, is something
we've run into with variable indexing, i.e., a double has 52 bit mantissa or
significand.  Hence, there is this 12-bit range where int-64 numbers lose
resolution when computed with double values.

The general case of mean() means this is complicated.  We are summing together
unknown N numbers, so we need ceil(log2(N)) bits overhead to ensure there is
no arithmetic overflow.  But in the case of median() it is just two numbers
that involve mean.  So, should we consider the special cases of median of
int64 and uint64?

For int64, given two numbers X and Y, if X and Y are of different sign, then
meanXY = (X+Y)/2.  If X and Y are the same sign, then meanXY = (2X - X + Y)/2
= X + (Y-X)/2, or something close ensuring rounding is appropriate.

For uint64, it's just meanXY = X + (Y-X)/2.

Keep in mind that this needs to be for vector/matrix data, so the test above
would involve logical arrays in some way.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54567>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/