Re: lexer.ll: Warn about non-UTF-8 characters (issue 5505090)

[Keith OHara]

On Sun, 01 Jan 2012 01:40:36 -0800, <address@hidden> wrote:

> Our lexer has been written with the decision of using non-compressed
> tables and without backing up.

Off topic, but maybe interesting to you: I don't think that decision was ever 
implemented.  I don't see any "%option full" or similar that would generate 
non-compressed tables.   For at least a few years lexer.ll did generate backup states; I 
avoided them as part of the purge of warnings.  Also, lexing takes a tiny fraction of 
LilyPond's execution time.   If you ever want to use a method that requires backing up, I 
think you should.

> I spent more than a day's worth on doing
> utf-8 right in the grammar.  That's pretty pointless.  It also means
> that we need to provide an error path for every item containing
> non-UTF-8 characters in order to get a UTF-8 related error message
> instead of something more mysterious.

Okay, then.
Consider a comment in your case/switch statement that points to some reference 
on the various types of UTF-8 validators.