Re: [bug-mes] mes lib: dtoab

[Danny Milosavljevic]

Hi Janneke,

the reason that dtoab doesn't work is an overflow in the long.  Did it work on 
i386?

IEEE 754 double precision has a 52 bit mantissa which means such a number would 
have 16 significant digits.  That cannot fit into long (10 significant digits), 
so the way dtoab is implemented is not correct.

(It's not as bad as it could be since dtoab handles the fractional part and the 
integral part separately)

In any case, maybe it's enough for our purposes.

The fix for ARM would be:

--- a/lib/mes/dtoab.c
+++ b/lib/mes/dtoab.c
@@ -29,7 +29,11 @@ dtoab (double d, int base, int signed_p)
   long i = (long) d;
   char *p = ntoab (i, base, signed_p);
   strcpy (dtoa_buf, p);
+#if __arm__
+  long f = (d - (double) i) * (double) 100000000;
+#else
   long f = (d - (double) i) * (double) 100000000000;
+#endif
   if (f)
     {
       if (f < 0)

If it was all right with you, we could just unconditionally make the precision 
even worse and just always use the shorter multiplicant :)

Otherwise we would have to have two different expected results in the tests and 
that's not nice.

Furthermore, I don't understand how it worked on i386, if it did.  Because 
sizeof(long) is 4 on i386 and 4 on arm.

pgpAiNNPk8K21.pgp
Description: OpenPGP digital signature