Re: [Orgmode] Precision setting for spreadsheet calculation

[Carsten Dominik]

On Feb 26, 2010, at 10:58 PM, Michael Brand wrote:

> Carsten Dominik wrote:
> > these are the internals of calc, about which I cannot do much.
>
> But
>
> | 0.16666666666666666667 |
> #+TBLFM: $1 = 1 / 2 * 3; p20 f20
>
> shows that the internals of calc are ok, isn't it?

Hi Michael,

you are completely right, I was wrong.

The other issue that is playing into this is that when
using a printf format specifier, the number is first
converted into a Lisp number, which is a double, with
16 significant digits...  At this step, the additional
digits are lost.

Maybe you find this interesting enough so that I can
persuade you to write a FAQ entry about these issues? :-)

- Carsten


> > Michael Brand wrote:
> > > But instead of
> > >
> > > | 0.166666666666666660 |
> > > #+TBLFM: $1 = 1 / 2 * 3; %.18f p20 f20
> > >
> > > which I get I would expect
> > >
> > > | 0.166666666666666667 |
> > > #+TBLFM: $1 = 1 / 2 * 3; %.18f p20 f20

- Carsten