[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Proposed interface changes for "parse.error custom"
From: |
Akim Demaille |
Subject: |
Re: Proposed interface changes for "parse.error custom" |
Date: |
Thu, 27 Feb 2020 09:11:11 +0100 |
Adrian,
> Le 26 févr. 2020 à 23:58, Adrian Vogelsgesang <address@hidden> a écrit :
>
> regarding our discussion on memory management for yyexpected_tokens
> --------------------------------------------------------------------------------------------------
>
>
>> If your use case in incrementally proposing possible keywords, then you want
>> them all, and a max of the possible number of tokens is available
>> statically: YYNTOKENS.
> You are right, I would want to retrieve the whole list of expected tokens.
>
>> I don't see where this would be useful.
>
> The code I am having in mind to do so would be
> int onstack_tokens[10];
> std::vector<int> heap_tokens;
> int cnt_tokens = context.yyexpected_tokens(onstack_tokens, 10);
> if (cnt_tokens > 10) {
> heap_tokens.resize(cnt_tokens);
> context.yyexpected_tokens(heap_tokens.data(), cnt_tokens);
> }
> int* tokens = (cnt_tokens <= 10) ? onstack_tokens : heap_tokens.data();
Yes, that definitely makes sense when no (reasonable) bound is known.
Actually if memory is really an issue, we could also use a bitset. But a list
of indexes seems the right approach here. An iterator over the set of expected
tokens would also work and meet all these use cases. But maybe that's
over-engineering.
Hum...
We should really think about that.
> That way, we can use the stack allocation in the common case (<= 10 expected
> tokens) and only fallback to the heap-allocation if we have too many expected
> tokens. At least in our grammar, it is common that only very few keywords are
> expected, except in a few places, where you can write almost everything. Now
> that I think about this, I am not actually sure how well this observation
> generalizes to other grammars, though. Preallocating YYNTOKENS (480 in our
> case)
Wow!!! I believe this is the largest set of tokens I heard of!
Congratulations :)
How many rules, how many states?
> on the stack would cost us ~2KB of stack space.
Right.
> Probably that would be fine...
>
>> The point here is efficiency
> It's funny how both of us are having performance/efficiency as a goal but are
> coming to a different conclusion :)
Yep :)
> I was not seeing your point about "not spend time counting how many more
> there are" on my own. Thanks for pointing that out! With that reasoning, I
> can completely agree that the existing implementation is fine.
Great, thanks!