Re: Suggestions for C++ parser generator

[Mark Boyall]

I've been playing with the Variant you posted me, and it's a big
improvement. I found it vastly easier to express complex logic in the
parser. I wanted to point out a couple of issues that I had:

Firstly, the Variant won't build on Visual Studio 2010, because the
assignment operator doesn't return a value. I simply added return *this; to
silence the compiler.

Secondly, Bison states that the default action for a rule is { $$ = $1; }.
However, this is not exactly true- when $1 is of a type that is implicitly
convertible to $$, for example via inheritance, Bison will issue a redundant
warning. Simply by adding { $$ = $1; } I invoked the implicit conversion and
removed the warning.

Thirdly, I attempted to update the Variant class to support move semantics
and there is an issue you should be aware of if you ever intend to support
them. The problem is that when using rvalue references, you have to pass the
type explicitly if you don't want it to be perfectly forwarded- and since
there are implicit conversions from a reference, this causes the type to be
deduced as a reference for perfect forwarding which cannot be placed in the
buffer. The solution here is to always use a factory function and always
explicitly pass the intended target type. I presume that, since older
compilers don't support rvalue references, if you were to choose to support
them it would have to be conditional flagged support.

For example,

    template <typename T>
    inline T&
    build (const T& t)
    {
      return *new (buffer.raw) T(t);
    }

should become

    template <typename T, typename Arg>
    inline T&
    build (Arg&& a)
    {
      return *new (buffer.raw) T(std::forward<Arg>(a));
    }

where T is always explicitly provided. Secondly, implicit constructions
should be removed- only empty Variants should be allowed to be constructed
and then build used on them.

On the plus side, it was mostly hassle-free to use rvalue references- as a
mutable reference is provided when using $1 for example then it was easy to
use std::move as appropriate.

On 28 July 2011 16:03, Mark Boyall <address@hidden> wrote:

>  Actually, the parser was not really meant to be subclassed.  Maybe that
>> was an error, agreed.
>
>
> In addition, the yyerror() function could be pure virtual. Adding functions
> into libraries like this is what virtual functions are for :) It would also
> alleviate the need for yylex() to take additional arguments in quite a few
> cases, because those arguments could be just data members of the derived
> class.
>
> The latest versions of Visual Studio and GCC both support move semantics,
> as does Clang and Comeau- basically all of the major compilers, as far as I
> know, now support rvalue references in their latest versions and sometimes
> going back a considerable way.
>
> Actually it is plain standard C++ (it's a deque), and it's useful.  What is
>> painful though is that it is a separate file.  I'd like to merge it into the
>> header file once for all (but backward compat will have to be maintained in
>> some way).
>
>
> Sure, but you'd still have to update it, and configure it, for if the
> compiler supports rvalue references. If you just used a deque directly then
> you would not have to worry about it.
>
>
>> Yes, indeed, but that's a whole new project you asking for here.  Bison is
>> a parser generator, not yet a parser-and-ast generator.
>
>
> That's probably true. I don't actually know that much about the Bison
> implementation- it's full of tables and stuff and I don't understand it
> tremendously well. What I really should have said is that I found it a
> slight pain when dealing with recursive rules to build data structures of
> those rules, which maintain the order they should do- especially when
> multiple recursive rules is invoked separately and repeatedly in the same
> construct. The problem with the existing functional programming style is
> that you can't pass anything *forwards*, only backwards- unless you're
> feeling brave with the minus numbers. One of the things I could suggest is
> instead of using goto to jump between states, you could use mutually
> recursive functions. The advantage of this is that firstly, each function
> can be strongly typed without the need for a hold-all union or variant, and
> secondly, each function can be given user-defined additional arguments on a
> per-function basis. Of course, I don't know how much effort it would be to
> enable such a design.
>
> The variant design you post about would be a massive improvement and solve
> a lot of my problems relating to using Bison. Right now I am using a
> (relatively) complex arrangement in my user-defined argument to hold the
> correct data structure for various recursive rules, and it's quite complex
> and hard to reason about. I'm using Bison 2.5 and don't see anything about a
> %variant option. I will look through the link you posted and see what I can
> make happen with it.
>
> On 27 July 2011 17:42, Akim Demaille <address@hidden> wrote:
>
>>
>> Le 26 juil. 2011 à 17:17, Mark Boyall a écrit :
>>
>> Hi!
>>
>> > I've got a couple of simple suggestions for the C++ parser generator.
>> >
>> > Firstly, yylex() should be declared as a (pure) virtual function in the
>> > parser interface. This would make using it re-entrantly significantly
>> easier
>> > and would mean that the user does not have to declare it.
>>
>> Actually, the parser was not really meant to be subclassed.  Maybe that
>> was an error, agreed.
>>
>> > Secondly, stack.hh should be replaced with the Standard stack class.
>> This
>> > will ensure that users who have move support can use move-only classes
>> when
>> > interacting with Bison.
>>
>> Actually it is plain standard C++ (it's a deque), and it's useful.  What
>> is painful though is that it is a separate file.  I'd like to merge it into
>> the header file once for all (but backward compat will have to be maintained
>> in some way).
>>
>> Where do you use move semantics?  Are you using some C++-11 compiler?
>>
>> > Thirdly, I suggest that Bison should generate it's own AST classes, with
>> > inheritance being used to link rules which have alternatives. This would
>> > solve, mostly, the problem of the semantic types only being POD data
>> types.
>>
>> Yes, indeed, but that's a whole new project you asking for here.  Bison is
>> a parser generator, not yet a parser-and-ast generator.
>>
>> Have you experienced the %variant mode in Bison development version?  It
>> works well, and avoids this nasty POD issue.  Have a look at
>> http://www.lrde.epita.fr/~akim/download/bison-2.4.570-7a582.tar.bz2.
>>  Look for variant in the documentation.  The following example shows what
>> you can do.  I need to polish and publish this :(  Help would be most
>> welcome.
>>
>> %type <::std::string> item;
>> %type <::std::list<std::string>> list;
>>
>> %%
>>
>> result:
>>  list  { std::cout << $1 << std::endl; }
>> ;
>>
>> list:
>>  /* nothing */ { /* Generates an empty string list */ }
>> | list item     { std::swap ($$, $1); $$.push_back ($2); }
>> ;
>>
>> item:
>>  TEXT          { std::swap ($$, $1); }
>> | NUMBER        { $$ = string_cast ($1); }
>> ;
>> %%
>>
>>
>>
>