[Octave-bug-tracker] [bug #58147] Matlab 2020 compatibility: dec2bin, dec2hex shouldn't error on negative inputs

[Nicholas Jankowski]

Follow-up Comment #5, bug #58147 (project octave):

here's a quick stab at a version of dec2bin that returns the 2's complement.
(not close to ready yet, but could use some algorithm pointers.)

before it calls dec2base, it uses:


  if (d < 0) ##only works for scalar numerical inputs. 
    if (-d <= 128)
      bitlen = 8; 
    elseif (-d <= 32768)
      bitlen = 16;
    elseif (-d <= 2^31)
      bitlen = 32;
    else
      ##TODO: extend bitlen to 64 for compatibility when bitcmp can handle
64bit
      bitlen = 53;
  endif
  
    d = bitcmp (-d, bitlen) + 1; ##returns same class as d, so ints get
limited near intmax?
  endif


bitcmp is limited to 53 bits, is not matlab compatible in general, and returns
the same class that is input, so run into some limits with large ints (i.e.,
intmax errors)

maybe there's a better way than bitcmp?

    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?58147>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/