token_buffer/object_base difficulty in parse_percent_token

[Juan Manuel Guerrero]

All versions of bison 1.3X that supply the regression test "invalid inputs"
fails on MSDOS/DJGPP randomly depending on the memory contents at run time.
If the test fails it will produce output similar to this one:

## -------------------------- ##
## GNU Bison 1.35 test suite. ##
## -------------------------- ##
53. regression.at:626: testing Invalid inputs...
regression.at:645: bison input.y
--- -   Sat Mar 30 11:17:52 2002
+++ /dev/c/_/srcs/gnu/bison-1.35/tests/testsuite.dir/at-stderr  Sat Mar 30 
11:17:52 2002
@@ -1,7 +1,7 @@
 input.y:2: invalid input: `?'
 input.y:3: invalid input: `}'
-input.y:4: invalid input: `%{'
-input.y:5: invalid input: `%&'
+input.y:4: invalid input: 
`%{\005\270\377\377\377\377[\211\354]\303\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\254
 \3
75\377\032 \222\001\036 &   ! \306\0020 `\0019 
R\001\375\377\375\377\375\377\375\377\030 \031 \034 \035 " \023 \024 
\334\002"!a\001: S\001\375\377\375\377x\001\036\001\321'
+input.y:5: invalid input: 
`%&\377\377[\211\354]\303\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\254
 \375\377\032 \222\
001\036 &   ! \306\0020 `\0019 R\001\375\377\375\377\375\377\375\377\030 \031 
\034 \035 " \023 \024 \334\002"!a\001: S\001\375\377\375\377x\001\036\001\321'
 input.y:6: invalid input: `%a'
-input.y:7: invalid input: `%-'
+input.y:7: invalid input: 
`%-\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\220\254
 \375\377\032 \222\001\036 &   ! \306\0020 `\
0019 R\001\375\377\375\377\375\377\375\377\030 \031 \034 \035 " \023 \024 
\334\002"!a\001: S\001\375\377\375\377x\001\036\001\321'
 
53. regression.at:626: FAILED near `regression.at:645'


The reason for this failure is that the token string stored in
char *object_base of struct token_obstack in function parse_percent_token
of file src/lex.c is never terminated with '\0'.
This makes the calling function read the contents of char *token_buffer
(a pointer to char *object_base) until it finds a '\0' somewhere in memory,
generating this difficult to reproduce runtime depending bug.
I have never observed this failure on linux but I am not familiar enough
with linux to fully understand this difference in testsuite behaviour between
linux and dos/djgpp.
Anyway, a patch similar to the one proposed by me below will be needed
to fix this issue for DJGPP.
Suggestions, objections, comments, etc. are welcome.
The patch is based on bison-1_29-branch.

Regards,
Guerrero, Juan Manuel



Index: ChangeLog
===================================================================
RCS file: /cvsroot/bison/bison/ChangeLog,v
retrieving revision 1.173.2.298
diff -u -r1.173.2.298 ChangeLog
--- ChangeLog   25 Mar 2002 10:08:18 -0000      1.173.2.298
+++ ChangeLog   30 Mar 2002 11:27:48 -0000
@@ -1,3 +1,8 @@
+2002-03-30  Guerrero, Juan Manuel  <address@hidden>
+
+       * src/lex.c (parse_percent_token): Zero terminate all token_obstack
+       and token_buffer strings.
+
 2002-03-25  Akim Demaille  <address@hidden>
 
        Version 1.35.
Index: src/lex.c
===================================================================
RCS file: /cvsroot/bison/bison/src/lex.c,v
retrieving revision 1.33.2.13
diff -u -r1.33.2.13 lex.c
--- src/lex.c   22 Jan 2002 10:29:28 -0000      1.33.2.13
+++ src/lex.c   30 Mar 2002 11:27:53 -0000
@@ -596,10 +596,12 @@
   switch (c)
     {
     case '%':
+      obstack_1grow (&token_obstack, '\0');
       token_buffer = obstack_finish (&token_obstack);
       return tok_two_percents;
 
     case '{':
+      obstack_1grow (&token_obstack, '\0');
       token_buffer = obstack_finish (&token_obstack);
       return tok_percent_left_curly;
 
@@ -607,28 +609,34 @@
         very ancient Yacc versions.  The paper of Johnson mentions
         them (as ancient :).  */
     case '<':
+      obstack_1grow (&token_obstack, '\0');
       token_buffer = obstack_finish (&token_obstack);
       return tok_left;
 
     case '>':
+      obstack_1grow (&token_obstack, '\0');
       token_buffer = obstack_finish (&token_obstack);
       return tok_right;
 
     case '2':
+      obstack_1grow (&token_obstack, '\0');
       token_buffer = obstack_finish (&token_obstack);
       return tok_nonassoc;
 
     case '0':
+      obstack_1grow (&token_obstack, '\0');
       token_buffer = obstack_finish (&token_obstack);
       return tok_token;
 
     case '=':
+      obstack_1grow (&token_obstack, '\0');
       token_buffer = obstack_finish (&token_obstack);
       return tok_prec;
     }
 
   if (!isalpha (c))
     {
+      obstack_1grow (&token_obstack, '\0');
       token_buffer = obstack_finish (&token_obstack);
       return tok_illegal;
     }