Re: Incorrect point-and-click URIs of files with non-ASCII content on Windows

[Patrick McCarty]

On 2009-11-02, Bertalan Fodor (LilyPondTool) wrote:
> Patrick McCarty wrote:
> >On 2009-11-01, Bertalan Fodor (LilyPondTool) wrote:
> >>> 3) UTF-8 characters.  In UTF-8 locales, terminals need to know about
> >>> the byte offset, so I am using the character count to specify this
> >>> offset.  An example would be "3:11:10".
> >>>
> >>>The third case is arguably misleading, so maybe it should be changed
> >>>to use the "3:10:10" instead.  I am okay with either format.  If we
> >>>want to use "3:10:10" instead, then an additional parameter would be
> >>>needed to calculate the byte offset.
> >
> >>I hardly believe anyone or anything should care about byte offset.
> >>LilyPond source files contain UTF-8 characters and not bytes! If a
> >>terminal/editor doesn't support UTF-8 character streams, than the
> >>terminal/editor should be fixed.
> >
> >Well, in its current state, LilyPond is not using "UTF-8 character
> >streams" for error message output and point-click URIs, so this would
> >likely require a rewrite.
> >
> >Do you have any suggestions about how to implement this in C++
> >(portably)?
>
> Unfortunately no. But that should not be related to the bad
> character offset problem.

I don't have a fix yet, but I think I discovered the problem.  Here
are the relevant lines from lily/source-file.cc:

  #if HAVE_MBRTOWC
        wchar_t multibyte[2];
        size_t thislen = mbrtowc (multibyte, line_chars, left, &state);
  #else
        size_t thislen = 1;
  #endif /* !HAVE_MBRTOWC */


On Windows, the variable `thislen' seems to always have the value `1',
so that's why a multibyte (UTF-8) character in a source file is
messing up the character/column count.

This means that either

  1) mbrtowc() on Windows always returns `1' regardless of the
  character it is considering.  This is doubtful.

  2) LilyPond's configure script is not detecting mbrtowc() when
  compiling the Windows installer with GUB, and so the `size_t
  thislen = 1;' line is executed instead.


The second case is much more likely.  Graham, I don't want to bug you
with this, but would you mind checking the log for mingw::lilypond to
see if configure detects the mbrtowc() function?  On my Linux system,
the output is

  checking for mbrtowc... yes


Thanks,
Patrick