Dynamic tokens

[Ervin Hegedüs]

Hi,

is there any way to make a parser with "dynamic" tokens?

I mean in compiling time I don't know the available tokens. Now I describe
the necessary token with regex, but I bumped into a problem.

The language syntax is some like this:

EXPRESSION: LANG_OP LANG_OP_ARGUMENT | LANG_OP_ARGUMENT

where (as you can see) the LANG_OP is optional. If there isn't LANG_OP,
that means that is the most usable operator (namely "@rx" in my case). The
syntax of the operator (with regex): "@[a-z][a-zA-Z0-9]+".

Example from the language:
@eq 1
@lt 2
@streq foo

The problem is that the LANG_OP_ARGUMENT could be anything - for example,
that could be also starts with "@". So, the next expression is valid:

@streq @streq

Now I'm using this rules:
@[a-z][a-zA-Z0-9]+     { BEGIN(ST_LANG_OP); return LANG_OP; }
<ST_LANG_OP>....

but now the operator isn't optional. If I write in the language:

"@rx" that means that's an operator argument, without operator.

I resolved this with put all available operators into one pattern, eg:
@(eq|lt|streq|...)    { BEGIN(ST_LANG_OP); return LANG_OP; }
...

and I can play with tokens and language rules... so it works as well.

I'ld like to modify the parser that it can be capable to extend to
recognize the operators (and other keywords of language) without modify and
rebuild the whole parser.

My idea is that if I can substitute the token pattern in the declaration
with a variable, and if I set up that variable before I call the parser. In
the example above, with a pseudo mode I could:

...
strcpy(VAR_LANG_OP_LIST, "@(eq|lt|streq|myop)");
...
yylex();

and in the declaration of token should be:

$VAR_LANG_OP_LIST   { BEGIN(ST_LANG_OP); return LANG_OP; }
....

Is there any way to solve this? If not, what do you think, how can I do
that?

Thank you,


a.