Re: Allow digits in identifiers (issue 6493072)

[k-ohara5a5a]

While we are thinking about this, I suggest we remove (later) the rule
forbidding backing-up states in the scanner.  It made only a 0.2% in the
worst-case scenario I could think of.

The rule had been violated, giving us the slightly slower scanner, for
about ten years (I estimate) prior to the warning-cleanup patch da949cdc
one year ago.

It costs a lot of programmer time to make the extra rules to save that
0.2%, and I know of no efficient cure for patterns that can require
arbitrary amounts of backing up.  \violin1_2_3_4_5


http://codereview.appspot.com/6493072/diff/5015/lily/lexer.ll
File lily/lexer.ll (left):

http://codereview.appspot.com/6493072/diff/5015/lily/lexer.ll#oldcode34
lily/lexer.ll:34: flex -b <this lexer file>
`-b, --backup, `%option backup''
     Generate backing-up information to `lex.backup'.  This is a list
of
     scanner states which require backing up and the input characters
on
     which they do so.  By adding rules one can remove backing-up
     states.  If _all_ backing-up states are eliminated and `-Cf' or
     `-CF' is used, the generated scanner will run faster (see the
     `--perf-report' flag).  Only users who wish to squeeze every last
     cycle out of their scanners need worry about this option.  (*note
     Performance::).

http://codereview.appspot.com/6493072/