[Octave-bug-tracker] [bug #43728] rounding error for integer range multiplied with floating point

[Markus Appel]

URL:
  <http://savannah.gnu.org/bugs/?43728>

                 Summary: rounding error for integer range multiplied with
floating point
                 Project: GNU Octave
            Submitted by: mappel
            Submitted on: Sun 30 Nov 2014 01:19:29 AM GMT
                Category: Interpreter
                Severity: 3 - Normal
                Priority: 5 - Normal
              Item Group: Incorrect Result
                  Status: None
             Assigned to: None
         Originator Name: 
        Originator Email: 
             Open/Closed: Open
         Discussion Lock: Any
                 Release: dev
        Operating System: GNU/Linux

    _______________________________________________________

Details:

In the following example, I would expect the superdiagonal to be 1
everywhere:


octave:16> m1=(-1:3)*0.1 ; m2=(0:4)*0.1 ; m1==m2'
warning: mx_el_eq: automatic broadcasting operation applied
ans =

   0   1   0   0   0
   0   0   1   0   0
   0   0   0   0   0
   0   0   0   0   1
   0   0   0   0   0


... but it isn't due to a floating point error:


octave:17> m1(4)-m2(3)
ans =    2.7756e-17


Seen on Ubuntu 12.04 with octave 3.8.0 and dev. Matlab R2012b gives the
expected result ( with bsxfun(@eq,m1,m2') instead of m1==m2' ).

Complementary tests:

It seems that something goes wrong when the range extends to negative
integers, I could not reproduce it with positive ranges only:


octave:89> m1=(1:5)*0.1 ; m2=(0:4)*0.1 ; m1==m2'
warning: mx_el_eq: automatic broadcasting operation applied
ans =

   0   0   0   0   0
   1   0   0   0   0
   0   1   0   0   0
   0   0   1   0   0
   0   0   0   1   0


And using a decimal that can be represented exactly also seems to work fine:


octave:94> m1=(-1:3)*0.125 ; m2=(0:4)*0.125 ; m1==m2'
warning: operator -: automatic broadcasting operation applied
ans =

   0   1   0   0   0
   0   0   1   0   0
   0   0   0   1   0
   0   0   0   0   1
   0   0   0   0   0






    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?43728>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/