Re: d: change the return value of yylex() from int to TokenKind

[Akim Demaille]

Hi Adela,

Sorry for the delays.

> Le 16 sept. 2020 à 20:47, Adela Vais <adela.vais99@gmail.com> a écrit :
> 
>> As a matter of fact, the TODO had a question:
>> 
>>   ** Change the return value of yylex
>>   Historically people were allowed to return any int from the scanner (which
>>   is convenient and allows `return '+'` from the scanner).  Akim tends to see
>>   this as an error, we should restrict the return values to TokenKind (not to
>>   be confused with SymbolKind).
>> 
>>   In the case of D, without the history, we have the choice to support or not
>>   `int`.  If we want to _keep_ `int`, is there a way, say via introspection,
>>   to support both signatures of yylex?  If we don't keep `int`, just move to
>>   TokenKind.
>> 
>> I was really curious to know if D's introspection made it possible to
>> support both signatures.  If it can't, or if we consider returning an
>> int is not right, then, sure, your commit (once the test suite issue
>> addressed) is the right path.

> D doesn't support this.

Ah, ok.  Maybe the scanner interface is not right, maybe it's too dynamic
and not static enough for the introspection to kick in early enough.

> Our reasoning was that int as a return value might be a bit confusing/unclear 
> for someone unfamiliar with the program, and TokenKind would give more 
> information about what is expected.

Yes, I agree with that.

> It is also Dlang good practice to choose the most specific type possible when 
> writing an interface, so TokenKind would be the best choice here.

Agreed.


However...

> I modified the failing test files. My understanding is that tokens given 
> using single quotes are not creating TokenKind entries, please correct me if 
> I am wrong. I separated the Bison declarations for D from the ones for C/C++, 
> because of the single quote declarations, as these tests were using the int 
> return value to its fullest, and changed the error tests' expected output for 
> the same reason. 

however I had not realized that this meant to make the calc.at test so much 
harder to read and maintain.  I have to think about that.

It's great that you found your way into tweaking the test suite, this will 
prove most useful in the future!

Cheers!

        Akim