Add support for unicode filenames on Windows (issue 206640043 by address@hidden)

[pkx166h]

Reviewers: ,

Message:
Pasting update in Tracker by David K for Masamichi's benefit:

--snip--

by address@hidden: Patch: Add support for unicode filenames on Windows
https://code.google.com/p/lilypond/issues/detail?id=4317

Ok, I've been stalling on this patch for about a week because I don't
have a good idea how to address it properly.  So let's start with the
worst first: at the current point of time, I don't see it making sense,
and in addition I will not be able to spend significant amount of time
addressing the problems arising in connection with it for about a month,
so I'd like a moratorium on it for about that time.

The main problem is that the next step needed for GUILEv2 migration
(which nobody is really interested in working on apart from myself) is
to get the coding mess sorted out.  GUILEv1 is basically working with
byte streams in its strings and string ports and files.  GUILEv2 is
coding system aware, uses either Latin-1 or UTF-32 as its string
internals, uses UTF-8 for string ports (necessitating copies and
conversions rather than being able to work in-place) and converts back
and forth all the time.

Since LilyPond has a lexer working on an UTF-8 coded byte stream and
data is liberally bounced around between files, strings, and string
ports, and the respective position pointers are not distinguished.

It does not help that GUILEv2 changes its ways to pick particular
encodings basically every 3 or 4 "stable" versions in the 2.0 series,
and 2.1 is different again.  So we need to carefully refactor the code
in order to have a chance of it working in both GUILEv1 and GUILEv2 (and
at all in GUILEv2).

This is an ugly and ungrateful task that has been at the front of my
this-is-really-up-next-so-let's-procrastinate-instead queue the last
month or so.  Letting a patch like this in right now would make the
situation worse.

Viewed realistically, this patch also needs Ghostscript updates in GUB
to make any sense.  There is no reason _those_ cannot go ahead first:
having a buffer of a few developer releases where we might discover and
sort out possible problems with newer versions of GhostScript on our
crosscompiled distributions seems sensible.

Personally, I don't really like the look of this patch at all as it is
very specifically useful only for one platform, and apart from that I'd
very much want to avoid seeing any wide character functions in LilyPond
code: that's rarely exercised library code of often dubious reliability
or availability, and most programmers are not overly comfortable
utilizing it.  I'd very much prefer that we'd find a better encapsulated
solution, probably by utilizing the encoding support of GUILEv2.  Which
again suggests that a moratorium on this patch makes sense, but that one
would not be "until David gets encodings sorted out so that we can
compile and test on GUILEv2" but rather on "until we are able to ditch
GUILEv1 altogether".  And given the current enthusiasm for working on
the GUILEv2 migration and the likelihood of necessary bug fixes in GUILE
itself becoming available as we discover problems, the latter is likely
quite longer than a month away.

The problem this patch tries to address is not a recent problem, and the
workaround ("don't use special Unicode characters in filenames") is
obvious.  So the urgency seems limited.  I very much agree that LilyPond
should not balk at file and directory names containing accented
characters on all platforms.  But I think we will be doing ourselves our
favor to postpone addressing this problem until we have moved LilyPond
to GUILEv2, and then look for the best-matching solution within _this_
framework.  Otherwise we complicate the coding issue work for GUILEv2
migration and that's something we can afford less than prolonging the
current filename encoding situation.

--
You received this message because you are the owner of the issue.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

Reply to this email to add a comment or make updates.

Description:
Add support for unicode filenames on Windows

On Windows, LilyPond can't handle
unicode filenames.

The patch replaces main(), and
hooks filename related functions.
This converts between UTF-16
unicode (Windows) and UTF-8 unicode
(LilyPond, libguile etc.).

LilyPond can handle unicode filenames
for *.ly, *.mid, *.ps.

*.pdf is not supported yet.
Ghostscript-8.70 that is included
in the binary distribution of LilyPond
can't handle unicode filenames.
This requires Ghostscript-9.10 or later

Please review this at https://codereview.appspot.com/206640043/

Affected files (+523, -0 lines):
  A flower/include/mingw-utf8.hh
  A flower/include/mingw-utf8-conv.hh
  A flower/include/mingw-utf8-func.hh
  A flower/include/mingw-utf8-hook.hh
  A flower/mingw-utf8-conv.cc
  A flower/mingw-utf8-func.cc
  A flower/mingw-utf8-hook.cc
  A flower/mingw-utf8-main.cc
  M lily/GNUmakefile
  M lily/main.cc