Starting States eat my bytes!

[ZuS]

I use exclusive starting states in flex to jump in and out of different sets
of tokens as I parse a somewhat complex binary input. Let me show this to
you visually. I will represent the yytext pointer with [].

0x53 [0xfb] 0x0a 0x01 0x02
As you can see, I just read the 0xfb byte. I know that bytes in bold belong
to a different context and I want to change the start state to parse them. I
also want to return a token to Bison representing 0xfb. So simplified code
looks something like this:
<CONTEXT1>"\xfb"              { BEGIN CONTEXT2; return FBTOKEN; }
<CONTEXT2>.                     { printf("Let's see what we got inside
yytext now: %x", (unsigned char)*yytext); }

The resulting print clearly shows that at entry into CONTEXT2, this was the
situation:
0x53 0xfb 0x0a [0x01] 0x02
Flex elegantly skips the 0x0a character, if it appears in the start a new
state, and only there. I don't want this behavior, so atm. I hack the s***
out of it:
<CONTEXT2>.               { 
                                           if ((unsigned int)(unsigned
char)yytext[-1] == 10)
                                                  yytext[0] = yytext[-1];
                                           printf("Let's see what we got
inside yytext now: %x", (unsigned char)*yytext);
                                      }
This works, but is disgusting. The [.] rule is the only one in CONTEXT2, so
I know 0x0a does not get eaten by a \n rule. I also know that this behavior
does not come up with other (haven't tested this exhaustively) binary
values, so for example:
0x53 [0xfb] 0x09 0x01 0x02               results in               0x53 0xfb
[0x09] 0x01 0x02
at the start of CONTEXT2.
I suppose this has something to do with YY_DO_BEFORE_ACTION or some other
definition, but I am not sure. I have a YY_USER_ACTION of my own, but it
really only sets yylloc stuff, so it's unlikely to be the problem. I am
looking through the verbose file to see if I can spot the issue, but that
takes a while. In the meantime can someone clue me in as to what the problem
is and what settings I can play with to manipulate this behavior?
-- 
View this message in context: 
http://www.nabble.com/Starting-States-eat-my-bytes%21-tp25577783p25577783.html
Sent from the Gnu - Bison - Help mailing list archive at Nabble.com.