[Octave-bug-tracker] [bug #54405] octave_idx_type index integer overflow math check doesn't work correctly

[Dan Sebald]

Follow-up Comment #5, bug #54405 (project octave):

OK thanks.  The ::error() variant is the one I want.

To change to sign.  I don't know, maybe it isn't so bad.  Can test by just
changing the sign and see how many BIST fail.  If it makes things easier to
deal with, it could be worth it.

So, with the ::error in place now, let me illustrate that in fact it is the
lost of precision because of the 53 bit mantissa that is being manifested
here.  I'll start out choosing the size to be the
std::numeric_limits<octave_idx_type>::max ().  That gets mapped to a negative
number (when it shouldn't).  Then I will repeat by replacing the least
significant digits with zero:


octave:6> fid = fopen("zeros1000by61.dat","r"); tic; xt = fread (fid,
[9223372036854775807, 9223372036854775807], 'char'); toc; fclose(fid);
OIT MAX: 9223372036854775807
nr: -9223372036854775808  nc: -9223372036854775808
(std::numeric_limits<octave_idx_type>::max () / nr): 0
nr * nc: 0
Elapsed time is 0.000118971 seconds.
octave:7> fid = fopen("zeros1000by61.dat","r"); tic; xt = fread (fid,
[9223372036854775800, 9223372036854775800], 'char'); toc; fclose(fid);
OIT MAX: 9223372036854775807
nr: -9223372036854775808  nc: -9223372036854775808
(std::numeric_limits<octave_idx_type>::max () / nr): 0
nr * nc: 0
Elapsed time is 0.000119925 seconds.
octave:8> fid = fopen("zeros1000by61.dat","r"); tic; xt = fread (fid,
[9223372036854775000, 9223372036854775000], 'char'); toc; fclose(fid);
OIT MAX: 9223372036854775807
nr: 9223372036854774784  nc: 9223372036854774784
(std::numeric_limits<octave_idx_type>::max () / nr): 1
nr * nc: 1048576
error: fread: dimension too large for Octave's index type
octave:8> 


One can see that in the 800 range the index starts to get mapped to the
correct int64 quantity.  So, 800, yeah I can believe that is roughly 9 or 10
bits loss in resolution.  So that is what's going on.

Regardless of the signedness of octave_idx_type, I would think going directly
to that from the index specification would be best.  That is, the parser can
detect a leading minus sign for indices and throw an error.  The parser can
also recognize the user typed a fractional portion in an index and given it is
non-negative, just toss those numerals after the period when it is an index. 
What the parser can't do is recognize the user might be using a double
variable as an index and there is a loss of resolution that comes with that. 
Internally one might be able to recognize such a thing is occurring, i.e.,
octave_idx_type greater than 2^53 (52?) from a double variable, and print out
a warning message about loss of resolution.  Does it matter that much? 
Probably not as its only these test cases with large-valued indices that
causes problems.  But still, I'm thinking if there is some way to simplify and
correct these corner cases, it could be worth it.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?54405>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/