Re: Encoding of LilyPond console output

[David Kastrup]

Wilbert Berendsen <address@hidden> writes:

> Everything boils down to the question in which encoding the 8bit
> console output of LilyPond is presented. I was always assuming that
> this was UTF-8, which worked correctly on Linux. I.e. both
> filenameswith accented letters and translated messages (such as
> French, with many accented letters) always showed up correctly in the
> LilyPond console output. Which is read by Frescobaldi as an 8bit
> bytestream and decoded into unicode strings using the UTF-8 encoding.
>
> Then, on Windows, I discovered that filenames do not use UTF-8
> encoding, but rather 'mbcs' or something like that (what is returned
> by sys.getfilesystemencoding() in Python).
>
> So I changed Frescobaldi to use that encoding when reading LilyPond
> console output, but then we discovered that translated messages (such
> as the French ones) with accented characters do show in a garbled
> encoding (clearly showing something like UTF-8 displayed as Latin1).
>
> So again I changed Frescobaldi, and now it reads the console output
> byte stream and parses that for file references (such as:
> file.ly:12:3: error: blabla) and decodes those filenames using the
> filesystem encoding, and the rest using UTF-8.
>
> This seems to work well: file references are correctly parsed and
> messages are readable still.

It will likely cease working at some point of time.  Lilypond is
supposed to be UTF-8.

> So everything thrown together: is my analysis of the mixed output
> encodings LilyPond uses on stdout and stderr correct?

Probably.

> And in line with this: can LilyPond be made more aware of this, and
> use the same encoding for all output (correctly encoding filenames)?

It will have to at one point of time.  I am currently preparing a patch
where it will reject all non-UTF-8 input, and of course this includes
filenames in \include and similar statements.  I have no idea how to
deal with the command line on Windows.

-- 
David Kastrup