Re: indentations instead of endtag in a grammar

[Conrad Irwin]

On 02/17/2010 02:22 PM, Andreas Wagner wrote:
> Thx for your fast answers. I will take a look.
> @ Istvan is it possible that u send me the *.l & *.y files for python
> (if u have them)?
> Or post a link. Always when i search for python and .y or bison i just
> find links to PyBison or PLY but no bison and lex file for Python.
> 

Real Python has a hand-coded tokenizer and their own parser generator,
probably explaining why you haven't found what you were looking for
(though downloading the source code would tell you that pretty quickly).

As part of a recent project, I implemented a lexer for Python [1]; as it
runs is a fairly complex environment (and I'm rubbish at writing C), you
won't be able to compile it, but the ideas are copyable. The bison
parser is auto-generated from some C++ class files, so it's not terribly
illuminating either, an excerpt:

if_statement: TOK_IF test_phrase TOK_COLON suite {
                $$ = IfStatement::parse($1, $2, $3, $4); }
            | if_statement TOK_ELSE TOK_COLON suite {
                $$ = IfStatement::parse($1, $2, $3, $4); }
            ;
suite: simple_statement TOK_NEWLINE {
        $$ = Suite::parse($1, $2); }
     | TOK_NEWLINE TOK_INDENT suite_body TOK_OUTDENT {
        $$ = Suite::parse($1, $2, $3, $4); }
     ;

Note that in addition to keeping a stack of indentation levels, the
lexer also keeps track of open brackets, as a newline inside a bracket
does not count as significant in Python (nor a newline after a \).

Yours
Conrad

[1] http://jelzo.com/got/181/parathon.l A lexer for Python(ish).