[Octave-bug-tracker] [bug #54567] median(uint8) gives bad results

[Dan Sebald]

Follow-up Comment #10, bug #54567 (project octave):

KT, here's a test of the unsigned 64-bit integer using min/max functions:


octave:110> x = nth_element (x64, 2:3, 2);
octave:111> min (x, [], 2) + (max (x, [], 2) - min (x, [], 2)) / 2
ans =

                     9
                    12
  18446744073709551608

[The following does not work generally]
octave:112> x(:,1) + (x(:,2) - x(:,1)) / 2
ans =

                     9
                    12
  18446744073709551608

octave:113> x(:,2) + (x(:,1) - x(:,2)) / 2
ans =

                    11
                    14
  18446744073709551610



I tested what happens when the unsigned "wrap" occurs.  Well, Octave is
programmed to not wrap.  I was curious what happens with wrapping because
sometimes one can devise a formula that allows integer wrapping so long as it
is guaranteed to unwrap.  Since Octave truncates rather than wraps, there's no
tricks that can be done.

OK, so let's think about an int64 formula then...

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54567>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/