Re: Changing the lookahead token

[Hans Aberg]

On 17 Sep 2012, at 11:15, David Kastrup wrote:

> Hans Aberg <address@hidden> writes:
> 
>> Hi David!
>> 
>> On 16 Sep 2012, at 10:51, David Kastrup wrote:
>> 
>>> But that's just guessing.  Are there any hard or soft criteria about
>>> when it may or may not be allowed to pull the lookahead token out from
>>> under Bison and put something else there?
>> 
>> You might look at the push parser, which allows one to call it
>> whenever a token is available. Then LALR(1) compacts the states so
>> that additional reductions may be called before an error is issued;
>> possibly something similar may happen in your case.
> 
> I am not saying that anything is happening in my case: I am _asking_
> what will be happening.  I am not keen on investing weeks of work
> reworking the grammar and then figuring out that my approach was doomed
> from the start.  It is clear that the lookahead token does not creep
> into the stack.  It is also clear that it is responsible for causing
> reductions, and the reductions have actions which I can use to swap the
> lookahead token.  

Looking back at your OP, the LALR(1) collects total information about the 
grammar and computes, via compaction, a series of states. From the user 
perspective, it is hard to know when the lookahead is used and not. The 
simplest way seems to be to do some experimentation, which I think is what the 
documentation possibly refers to, but that might be too volatile for your 
application.

>> One way around it is to try to rewrite the grammar, so that the tokens
>> are not needed.
> 
> Not possible since the respective arguments are not delimited and may
> continue indefinitely.
> 
> c   c-.   c-.-^
> 
> are all valid music arguments.  So I need the lookahead token to decide
> where the argument may end, and the lexical class of the result is only
> decided upon executing user-definable code.  So I basically need to push
> information before the lookahead token.  I have a stack for that in the
> lexer and already use it in cases where I don't have a lookahead token.
> The question is whether I can push the lookahead token there when it has
> already triggered reductions.

As for this last question, I think it will screw up the parsing in rather 
unexpected ways. 

Have you considered the GLR parser? If the correct parse depends on a future 
lookahead, it will split and join when the correct parse is used.

>> Another might be to use LR(1) which makes sure actions are applied as
>> dictated by the tokens.
> 
> I want to avoid going there.  

One just sets a %define - see the manual.

Hans