write past end of buffer in vasnprintf() implementation of %f

[Ben Pfaff]

When I apply the following patch to gnulib:

----------------------------------------------------------------------
diff --git a/tests/test-vasnprintf.c b/tests/test-vasnprintf.c
index 19731bc93378..105ac24c94a3 100644
--- a/tests/test-vasnprintf.c
+++ b/tests/test-vasnprintf.c
@@ -59,6 +59,11 @@ test_function (char * (*my_asnprintf) (char *, size_t *, 
const char *, ...))
       if (result != buf)
         free (result);
     }
+
+  size_t length = 8;
+  char *result = my_asnprintf (buf, &length, "%2.0f", 0x1.e38417c792296p+893);
+  if (result != buf)
+    free (result);
 }
 
 static char *
----------------------------------------------------------------------

and then run:

CC='gcc -fsanitize=address -g -O0' ./gnulib-tool --test vasnprintf 
vasnprintf-posix

I get the following failure from test-vasnprintf:

----------------------------------------------------------------------
==17880==ERROR: AddressSanitizer: heap-buffer-overflow on address 0xf4b03ece at 
pc 0xf770f8ed bp 0xffac9638 sp 0xffac962c
WRITE of size 1 at 0xf4b03ece thread T0
    #0 0xf770f8ec in convert_to_decimal ../../gllib/vasnprintf.c:899
    #1 0xf770f8ec in scale10_round_decimal_decoded ../../gllib/vasnprintf.c:1292
    #2 0xf771057c in scale10_round_decimal_double ../../gllib/vasnprintf.c:1328
    #3 0xf771755c in vasnprintf ../../gllib/vasnprintf.c:4119
    #4 0xf770c692 in my_asnprintf ../../gltests/test-vasnprintf.c:76
    #5 0xf770ca39 in test_function ../../gltests/test-vasnprintf.c:64
    #6 0xf770c384 in test_vasnprintf ../../gltests/test-vasnprintf.c:84
    #7 0xf770c384 in main ../../gltests/test-vasnprintf.c:96
    #8 0xf6f2be80 in __libc_start_main (/lib/i386-linux-gnu/libc.so.6+0x18e80)
    #9 0xf770c4de  
(/home/blp/pspp/gnulib2/testdir21347/build/gltests/test-vasnprintf+0x14de)

0xf4b03ece is located 0 bytes to the right of 270-byte region 
[0xf4b03dc0,0xf4b03ece)
allocated by thread T0 here:
    #0 0xf71cc334 in malloc (/usr/lib/i386-linux-gnu/libasan.so.4+0xe0334)
    #1 0xf770f223 in convert_to_decimal ../../gllib/vasnprintf.c:863
    #2 0xf770f223 in scale10_round_decimal_decoded ../../gllib/vasnprintf.c:1292

SUMMARY: AddressSanitizer: heap-buffer-overflow ../../gllib/vasnprintf.c:899 in 
convert_to_decimal
Shadow bytes around the buggy address:
  0x3e960780: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e960790: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e9607a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e9607b0: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x3e9607c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x3e9607d0: 00 00 00 00 00 00 00 00 00[06]fa fa fa fa fa fa
  0x3e9607e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e9607f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e960800: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e960810: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x3e960820: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==17880==ABORTING
FAIL test-vasnprintf (exit status: 1)
----------------------------------------------------------------------

The line in convert_to_decimal() cited above is the assignment here:

      /* Terminate the string.  */
      *d_ptr = '\0';

I guess that the space calculation passed to malloc() at the top of the
same function is not precise.  I don't know whether the right thing to
do is to just add one.

This bug was originally reported against GNU PSPP:
        https://savannah.gnu.org/bugs/?func=detailitem&item_id=54686

For this report, I've simplified it to remove the PSPP dependency (and
to make sure it isn't somehow a PSPP bug).

I'd appreciate it if someone more familiar with this part of the
vasnprintf() code, which seems rather opaque to a newcomer, would
investigate this.  Bruno, you're the main committer to this file, so I'm
guessing that's you, hence the CC.

Thanks a lot!