Finding out when a token is consumed

[Frank Heckenbach]

I have a bison parser, and I'd like to do something whenever a token
is consumed (used in a rule). More precisely, before an action is
run, something should be done for each terminal used in the rule.

Obviously, when I handle the tokens when they're lexed, that's too
early in case of parser lookahead. OTOH, if I handle the previous
token at the start of the lexer before the next one is lexed, that's
too late if there was no lookahead. So a pure lexer-based solution
doesn't seem to work.

Putting the extra code into every action would be clumsy. So I think
I need the possibility to run some code before any action (something
like YY_USER_ACTION in flex).

I see that YYLLOC_DEFAULT does this -- and the following example
using it seems to do what I want (and perhaps explains better what I
mean if my description isn't clear). But it seems like abuse of the
location tracking feature. Is there any other/better way?

%{
#include <stdio.h>
#include <stdlib.h>

static void yyerror (const char *s)
{
  printf ("%s\n", s);
}

static int yylex (void);

static void output_last (const char *);

#define YYLLOC_DEFAULT(A, B, C) \
  if (yychar == YYEMPTY) output_last ("parser => no lookahead");
%}

%token NUM

%locations

%%

expression:
    term
  | expression '+' term
      { $$ = $1 + $3; printf ("%i + %i = %i\n", $1, $3, $$); }
  ;

term:
    factor
  | term '*' factor
      { $$ = $1 * $3; printf ("%i * %i = %i\n", $1, $3, $$); }
  ;

factor:
    NUM
  | '(' expression ')'
      { $$ = $2; }
  ;

%%

static int last = -1;

static void output_last (const char *n)
{
  if (last >= 0)
    {
      if (last == NUM)
        printf ("consumed NUM %i", yylval);
      else
        printf ("consumed %c", last ? last : '$');
      printf (" (called from %s)\n", n);
    }
  last = -1;
}

static int yylex ()
{
  int c;
  output_last ("lexer => lookahead");
  c = getchar ();
  if (c == EOF || c == '\n')
    c = 0;
  if (c >= '0' && c <= '9')
    {
      yylval = c - '0';
      c = NUM;
    }
  if (c == NUM)
    printf ("lexed NUM %i\n", yylval);
  else
    printf ("lexed %c\n", c ? c : '$');
  last = c;
  return c;
}

int main ()
{
  return yyparse ();
}

Sample session:

# echo '1+2*3+4' | ./token-consume-test
lexed NUM 1
consumed NUM 1 (called from parser => no lookahead)
lexed +
consumed + (called from lexer => lookahead)
lexed NUM 2
consumed NUM 2 (called from parser => no lookahead)
lexed *
consumed * (called from lexer => lookahead)
lexed NUM 3
consumed NUM 3 (called from parser => no lookahead)
2 * 3 = 6
lexed +
1 + 6 = 7
consumed + (called from lexer => lookahead)
lexed NUM 4
consumed NUM 4 (called from parser => no lookahead)
lexed $
7 + 4 = 11

(E.g., the second `+' is lexed before the first addition is done,
but consumed afterwards, as is typical with lookahead tokens.)

Frank

-- 
Frank Heckenbach, address@hidden
http://fjf.gnu.de/
GnuPG and PGP keys: http://fjf.gnu.de/plan (7977168E)