|
From: | Robert Jenssen |
Subject: | [Octave-bug-tracker] [bug #56936] format short displays 6, rather than 5, significant digits as documented |
Date: | Sun, 5 Apr 2020 19:25:59 -0400 (EDT) |
User-agent: | Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36 |
Follow-up Comment #5, bug #56936 (project octave): I know it is pointless saying that MATLAB is wrong but in this case MATLAB is wrong! Left-hand zeros are not significant. The correct result to 5 significant figures is "0.33333". For the default format short, output_precision(5), the number of significant digits shown varies between 4 and 7: octave:1> version ans = 7.0.0 octave:2> format short octave:3> output_precision ans = 5 octave:4> 0.00123456 ans = 1.2346e-03 octave:5> 0.0123456 ans = 0.012346 octave:6> 0.123456 ans = 0.1235 octave:7> 1.23456 ans = 1.2346 octave:8> 12.3456 ans = 12.346 octave:9> 123.456 ans = 123.46 octave:10> 1234.56 ans = 1234.6 octave:11> 12345.6 ans = 1.2346e+04 octave:12> 123456 ans = 123456 octave:13> 1234567 ans = 1234567 octave:14> 12345678 ans = 1.2346e+07 _______________________________________________________ Reply to this item at: <https://savannah.gnu.org/bugs/?56936> _______________________________________________ Message sent via Savannah https://savannah.gnu.org/
[Prev in Thread] | Current Thread | [Next in Thread] |