Re: Token types with constructor

[Martin Trautmann]

Hi Hans,

I think I understood the main problem. But can you tell me what a bison
parser may do with tokens? Am I right with:
1) flex creates the token object (no problem: constructor is called)
2) bison stores it in a stack *++yyvsp = yylval;
Both variables are of type YYSTYPE, so operator=() is called
3) when a stack overflow occurs: bison copies the stack
(!!! no copy Constructor / operator= is called !!!)
4) bison calls a rule function that gets a stack element as parameter
This access is done using the correct type and shouldn't do any harm.

And there is one version of YYCOPY:
      do                                        \
        {                                       \
          register YYSIZE_T yyi;                \
          for (yyi = 0; yyi < (Count); yyi++)   \
            (To)[yyi] = (From)[yyi];            \
        }                                       \
      while (0)
That would have no problems using operator=() as well. So the only
additional thing to do would be to force the usage of that.

But I think in my small parser with the token type I showed you, stack
overflows didn't occur during my tests.
And in my other parser I do a special version of reference counting:
"object there or not there":
class Token
{
  int type;
  u{
    T1* t1;
    T2* t2;
        ...
  };
  T1& t1();
  T2& t2();
        ...
  Token() : type(no_token) {}
};

The only problem is a memory leak, that I don't free any Tokens finally
on the stack.

Oh, Finally I found a real problem:
Token::operator=() can't be sure whether the destination object was
already initialized by a constructor or not.

So there are to ways:
1) YYSTACK_ALLOC constructs all tokens on the stack
2) back to the roots: how does plain C work? ;-)

Do you know any parser that stores more than just some integers and is
implemented in a clean way?

bye

Martin

Am Dienstag, den 31.08.2004, 19:35 +0200 schrieb Hans Aberg:
> At 09:55 +0200 2004/08/30, Martin Trautmann wrote:
> >> This is an easy fix. But a problem is that you are using the C-stack, and
> >> that does not invoke the C++ copy constructors when reallocating. So funny
> >> things will probably start to happen then.
> >>
> >But if you name the generated parser file parser.cc and compile it with
> >g++ it makes no problem for me.
> >The C++ Compiler should support all C features in the rest of the parser
> >as well.
> >
> >I attached my simple parser that uses my solution. And from time to time
> >this parser compiled fine with the official bison versions. Of course
> >this simple parser wouldn't really need that constructor stuff. But I
> >have also a much more complex parser whose token defines pointers to all
> >possible types in a union (to save memory) but access functions allow an
> >access as easy as this would be with a struct.
>
> The C parser may well compile under C++, but it will not invoke the C++
> copy constructors when reallocating its parser stack, so your C++ code may
> break then. If your YYSTYPE is POD, then you are safe though, because then
> it will behave as under C.
>
> So if you use:
>   struct Token
>   {
>     std::string string;
>     union{
>       double scalar;
>       bool boolean;
>       Expression *exp;
>     }u;
>   };
> and your parser stack reallocates, you may get funny strings or a memory
> error (segmentation fault).
>
> I use:
> class semantic_type {
> public:
>   long number_;
>   std::string text_;
>   object object_;
>
>   semantic_type() : number_(0) {}
> };
>
> #define YYSTYPE semantic_type
>
> And here, the class object maintains a reference count to a polymorphic C++
> class hierarchy. But then I use, for parser stack, a standard C++
> sequential container. If you are going to use std::string in your type,
> then you can just as well do this, too: There is no particular gain in
> using an pointer xpe, as you do then, if you already have a non-POD data
> type in your YYSTYPE.
>
>   Hans Aberg
>
>
--