Re: extra keys from XS parser not decoded?

[Patrice Dumas]

On Fri, Feb 25, 2022 at 07:06:19AM +0000, Gavin Smith wrote:
> On Thu, Feb 24, 2022 at 11:50:51PM +0100, Patrice Dumas wrote:
> > Hello,
> > 
> > I have some differences between the perl Parser and the XS parser for
> > non ascii strings for extra values.  The perl Parser seems to be ok.
> > 
> > My feeling is that the strings are not properly decoded.  An analysis,
> > that may be completly off, is that SvUTF8_on should be called on
> > something, maybe around l 479 in tp/Texinfo/XS/parsetexi/api.c:
> > 
> >             case extra_string:
> >               { /* A simple string. */
> >               char *value = (char *) f;
> >               STORE(newSVpv (value, 0));
> >               break;
> >               }
> > 
> 
> I think you're right.  Any time a string is created it should probably
> have the UTF-8 flag set, unless it represents a filename.

It is ok too, I think, if we know that it can only be ascii, like
@-command names or type.  I guess that there should be strings in indices
structures too.

If it represents a filename, there may be a need to recode to the
encoding filenames are encoded to, as is done for @include file names.

> > Also maybe there could be a list to avoid decoding strings that should
> > not be decoded.  For now I spotted one in the perl Parser, but which is
> > not in the final tree for now (extra->file for include).
> > 
> > -- 
> > Pat
> >