Parsing included files, part II.

[Ervin Hegedüs]

Hi all,

(sorry, a bit long and difficult description...)


few days ago I asked a question about config file inclusions:
https://lists.gnu.org/archive/html/help-bison/2020-08/msg00007.html

and I wrote that I solved that:
https://lists.gnu.org/archive/html/help-bison/2020-08/msg00012.html

I rewrited the code, now I can include files on unlimited levels,
because I switched to dynamic memory handling. Everíthing works
as well - if I use the "include" directive sequentially.

Now I'ld like to move on, and try to implement the "include
patterns", eg.:

include foo*.conf

Here are the relevant part of my code:

in my lexer:
============
<INITIAL>include                                  { 
BEGIN(ST_INCLUDE_DIRECTIVE); push_sym(); return T_INCLUDE_DIRECTIVE; }
<ST_INCLUDE_DIRECTIVE>[0-9A-Za-z_\/\.\-\*\:]+     { include_stack_ptr++; 
push_sym(); parse_config(yytext); BEGIN(INITIAL); return 
T_INCLUDE_DIRECTIVE_ARGUMENT; }
<ST_INCLUDE_DIRECTIVE>\"[0-9A-Za-z_\/\.\-\*\:]+\" { include_stack_ptr++; 
push_sym(); parse_config(yytext); BEGIN(INITIAL); return 
T_INCLUDE_DIRECTIVE_ARGUMENT_QUOTED; }
<<EOF>>       {
                printf("Got EOF, stack_ptr: %d\n", include_stack_ptr);
                BEGIN(INITIAL);
                if (include_stack_ptr >= 0) {
                    close_and_cleanup(0);
                } else {
                    yyterminate();
                }
              }

my parser:
==========
config_include:
      T_INCLUDE_DIRECTIVE T_INCLUDE_DIRECTIVE_ARGUMENT         { printf("%s 
%s\n", $1, $2); free($1); free($2); }
    | T_INCLUDE_DIRECTIVE T_INCLUDE_DIRECTIVE_ARGUMENT_QUOTED  { printf("%s 
'\"%s\"'\n", $1, $2); free($1); free($2); }
;

notes:
* push_sym() just push the recognized token to stack, so I can
  use that in parser:

  yylval.s = strdup(yytext);

* include_stack_ptr points to top of different stacks, eg. files,
  buffers, ...

* parse_config() gets the filename, expands it to full path, and
  tries to handle it with wordexp(3), like this:

    wordexp(exp_fullname, &p, 0);
    w = p.we_wordv;
    for (size_t wi = 0; wi < p.we_wordc; wi++) {
        ... few error handling...
        open_and_parse(...)

* the open_and_parse() tries to open the file, makes some
  modifications into inputbuff (a local buffer), and starts to
  parse:

        if (include_stack_ptr == 0) {
            yy_scan_string(inputbuff);
            yyparse();
        }
        else {
            include_stack[include_stack_ptr-1] = YY_CURRENT_BUFFER;
            yy_scan_string(inputbuff);
        }

* there is the close_and_cleanup(), which closes the opened
  files, cleans up the not-used stack items, and so on...
  This routine is called when <<EOF>> detected.

Here are two examples:

$ ls -1 testinc/
foo1.conf
foo2.conf
incfoo1.conf
incfoo2.conf

$ cat testinc/foo1.conf 
# file: foo1.conf
$ cat testinc/foo2.conf 
# file: foo2.conf
$ cat testinc/incfoo1.conf 
include foo1.conf
include foo2.conf

and finally:
$ cat testinc/incfoo2.conf
include foo*.conf

So, my code works when I try to parse the incfoo1.conf, which
contains the included files sequentially:

Opening file: /home/airween/src/parsertest/testinc/incfoo1.conf
Opening file: /home/airween/src/parsertest/testinc/foo1.conf
include foo1.conf     <- this lines is created by the parser, see below
THIS IS A COMMENT: '# file: foo1.conf'   <- this is the content of file
Got EOF, stack_ptr: 1  <- IMPORTANT, here is the EOF
Closing file: /home/airween/src/parsertest/testinc/foo1.conf, stack_ptr: 1
Opening file: /home/airween/src/parsertest/testinc/foo2.conf
include foo2.conf     <- also by the parser
THIS IS A COMMENT: '# file: foo2.conf'
Got EOF, stack_ptr: 1
Closing file: /home/airween/src/parsertest/testinc/foo2.conf, stack_ptr: 1
Got EOF, stack_ptr: 0  <- end of incfoo1.conf
Closing file: /home/airween/src/parsertest/testinc/incfoo1.conf, stack_ptr: 0
Parser finished

Now here is the output when I call it with incfoo2.conf, which
contains "include foo*.conf":

Opening file: /home/airween/src/parsertest/testinc/incfoo2.conf
Opening file: /home/airween/src/parsertest/testinc/foo1.conf
Opening file: /home/airween/src/parsertest/testinc/foo2.conf <- wait, where is 
the EOF?
include foo*.conf   <- here is the output of parser
THIS IS A COMMENT: '# file: foo2.conf'    <- uhm, this is the last file
Got EOF, stack_ptr: 1    <- oh, here is the EOF of last file
Closing file: /home/airween/src/parsertest/testinc/foo2.conf, stack_ptr: 1
THIS IS A COMMENT: '# file: foo1.conf'    <- ops, this is the *first* file
Got EOF, stack_ptr: 0    <- ahm, and this is the first EOF
Closing file: /home/airween/src/parsertest/testinc/incfoo2.conf, stack_ptr: 0

as you can see:
* the order of files are reversed
* at least one file closing is missing (because one EOF missing)



Do you have any idea, why can't realize the lexer the <<EOF>> of
first included file? And why had changed the order of files?

I know this is a very complicated code and description, but I
hope somebody made a similar code, and can help me :).


Thank you,


a.