octave-bug-tracker
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Octave-bug-tracker] [bug #63848] Mean: some single class inputs can hit


From: Dmitri A. Sergatskov
Subject: [Octave-bug-tracker] [bug #63848] Mean: some single class inputs can hit precision limit with sum
Date: Wed, 1 Mar 2023 10:43:54 -0500 (EST)

Follow-up Comment #9, bug #63848 (project octave):

It is a saturation at flintmax('single') = 16777216.
With double it is 9007199254740992

There are better summations algorithms. 
See lots of discussions and proposals in

https://savannah.gnu.org/bugs/?61143

Dmitri.
-- 



    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?63848>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/




reply via email to

[Prev in Thread] Current Thread [Next in Thread]