[Octave-bug-tracker] [bug #53199] hist returns different results depending on 2nd 1:n/n input argument

[Michael Leitner]

Follow-up Comment #7, bug #53199 (project octave):

I do not agree with this analysis. The relevant part of the code is lines
108--109 in hist.m. For n=55, min_val=0 and max_val=55 you would get in
perfect arithmetic the range (0.5:55). With floating-point arithmetic you get
deviations


((.5:55)/55)*55-(.5:55)


This is due to the division and subsequent multiplication by 55, which as
intermediate result gives numbers that cannot be exactly expressed (the final
result of course is expressible). I have deviations at entry 4, 8, 15 and 16.
In all but the third case these are half the value of the corresponding
epsilon eps(4), eps(8), eps(16), that is, the resulting number is e.g.
15.99999.... (which has an eps that is half the one of 16). So these are not
large deviations, but have to be expected (in fact I am surprised that the
result is always so accurate, in most cases even exact). 

Of course you could change the order of expressions and fuse the division in
line 108 and the multiplication in line 109 and only afterwards multiply with
the range. Perhaps this is the actual code Matlab uses, so this would then
give (on the same architecture!) results that are perfectly compatible with
Matlab. It would not hurt to make this change. But actually, this problem is
due to the user misunderstanding the arguments of hist() -- with a sample of
integers from 0 to 55 you would of course want 56 bins, not 55. Using 55 bins
lets all values fall exactly on bin boundaries, so you will be susceptible to
rounding errors, no matter what you do. 

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?53199>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/