Re: Dynamic tokens

[Hans Åberg]

> On 2 Feb 2020, at 20:29, Ervin Hegedüs <address@hidden> wrote:
> 
> is there any way to make a parser with "dynamic" tokens?
> 
> I mean in compiling time I don't know the available tokens.

It is not possible to have dynamically created token values, …

> Now I describe
> the necessary token with regex, but I bumped into a problem.

… but it might be possible to use augmented methods.

> The language syntax is some like this:
> 
> EXPRESSION: LANG_OP LANG_OP_ARGUMENT | LANG_OP_ARGUMENT
> 
> where (as you can see) the LANG_OP is optional. If there isn't LANG_OP,
> that means that is the most usable operator (namely "@rx" in my case). The
> syntax of the operator (with regex): "@[a-z][a-zA-Z0-9]+".
> 
> Example from the language:
> @eq 1
> @lt 2
> @streq foo
> 
> The problem is that the LANG_OP_ARGUMENT could be anything - for example,
> that could be also starts with "@". So, the next expression is valid:
> 
> @streq @streq

So here you might have a context switch that is set when the operator token 
comes, that says that the next token, even if it is a valid operator name, 
should be treated as an argument. It when the argument is finished, set the 
switch back.

> Now I'm using this rules:
> @[a-z][a-zA-Z0-9]+     { BEGIN(ST_LANG_OP); return LANG_OP; }
> <ST_LANG_OP>....
> 
> but now the operator isn't optional.

Something must follow in the grammar, so the switch may be set back in the 
grammar. Check in the .output file for clues.

> If I write in the language:
> 
> "@rx" that means that's an operator argument, without operator.

One can have a symbol table that stores all operator names. If it is not there, 
return it as ain identifier. This way, one can dynamically define new operator.

If further, the table stores the token value, it can be used for other object, 
like variables that may have different syntax depending on type.