Re: Fwd: Q: processing multiple mixed data streams using bison

[Laurence Finston]

> Thanks for your reply. I did receive it from the list. I cannot afford
> to use threads as I am processing tens of thousands of clients at the
> same time.

> So I have to take care of errors while doing this...that makes it
> complicated but I don't have a choice :(

What you want to do sounds to me like implementing "context switches"
within Bison.  I don't think Bison is designed to be used in that way
and I don't think it's a promising approach.  However, that's just my 
opinion.  I do think there's probably an easier way of doing the job, 
though.

If you've got that many clients, I think it would be a good idea to use 
as many threads as you can, even if you can't allocate one to 
each client.  You can keep track of the number of threads you've spawned and
make sure you don't exceed the maximum number.  If you can't use threads, then
I would recommend using processes.  I think using a single instance of
'yyparse()' would be unnecessarily slow.

If your grammar has a structure similar to the following, I think it 
should be possible to interleave requests from different sources:

program: statement_list END

statement_list: statement_list statement

statement:   print_request
           | database_request
           | distributed_processing_request
           | (etc.)

This way, after every 'statement', you more-or-less "start fresh".
Then you would just need to arrange to queue the requests.  This is, however,
not the parser's problem.  If you don't need to keep information pertaining to
the requests after they've been processed, great.  If you do, you can use the
parameter passed to 'yyparse()' for doing so.  It might be a good idea to
maintain a separate object for each client and use the parameter to point to
the object associated with the current client.  In my parser, the objects
pointed to by the parameter represent the state of the parser.  If you choose
to try to implement a solution along these lines, you would need one for each
client, but the basic idea is the same.  If you're interested, you might want
to have a browse through the archives, because I've talked about this way of
using the parameter before.  In particular, you would not need to make any
changes to Bison itself, which I think would be a great advantage.

> There is no C++ version of bison...is there? 

You can generate a C++ parser class.  The last time I checked, the parser
function generated when doing this was not reentrant.  It may be now, I don't
know.  I don't use it myself.  I have Bison generate a (reentrant) C function,
use C++ in the actions, and compile with 'g++'.  I have no complaints
whatsoever about the results.

> Flex has one and it seems
> all I have to do is save and restore the class as all state of the
> lexer in inside the class.

I had problems using the C++ scanner class generated by Flex.  In my project,
I do use Flex, but not in combination with Bison.  I don't remember whether
the C version of 'yylex()' generated by Flex is reentrant.  I believe it is in
what was then a beta version, but I didn't want to use it.  You might want to
consider writing your own version of 'yylex()'.  It's not that hard, and you
might find it easier to implement the queueing of the requests from your
numerous clients.

In the approach I've outlined, you wouldn't need to save and restore anything.
  You would just need to point a pointer at different objects at different
times.  You would pass the same parameter to 'yylex()' as you do to
'yyparse()'.

Laurence Finston
http://www.gnu.org/software/3dldf/LDF.html