Re: Add support for unicode filenames on Windows (issue 206640043 by address@hidden)

[Masamichi HOSODA]

Thank you for the mail.
I've written reply to the tracker web page.

https://code.google.com/p/lilypond/issues/detail?id=4317

> Reviewers: ,
> 
> Message:
> Pasting update in Tracker by David K for Masamichi's benefit:
> 
> --snip--
> 
> by address@hidden: Patch: Add support for unicode filenames on Windows
> https://code.google.com/p/lilypond/issues/detail?id=4317
> 
> Ok, I've been stalling on this patch for about a week because I don't
> have a good idea how to address it properly.  So let's start with the
> worst first: at the current point of time, I don't see it making
> sense,
> and in addition I will not be able to spend significant amount of time
> addressing the problems arising in connection with it for about a
> month,
> so I'd like a moratorium on it for about that time.
> 
> The main problem is that the next step needed for GUILEv2 migration
> (which nobody is really interested in working on apart from myself) is
> to get the coding mess sorted out.  GUILEv1 is basically working with
> byte streams in its strings and string ports and files.  GUILEv2 is
> coding system aware, uses either Latin-1 or UTF-32 as its string
> internals, uses UTF-8 for string ports (necessitating copies and
> conversions rather than being able to work in-place) and converts back
> and forth all the time.
> 
> Since LilyPond has a lexer working on an UTF-8 coded byte stream and
> data is liberally bounced around between files, strings, and string
> ports, and the respective position pointers are not distinguished.
> 
> It does not help that GUILEv2 changes its ways to pick particular
> encodings basically every 3 or 4 "stable" versions in the 2.0 series,
> and 2.1 is different again.  So we need to carefully refactor the code
> in order to have a chance of it working in both GUILEv1 and GUILEv2
> (and
> at all in GUILEv2).
> 
> This is an ugly and ungrateful task that has been at the front of my
> this-is-really-up-next-so-let's-procrastinate-instead queue the last
> month or so.  Letting a patch like this in right now would make the
> situation worse.
> 
> Viewed realistically, this patch also needs Ghostscript updates in GUB
> to make any sense.  There is no reason _those_ cannot go ahead first:
> having a buffer of a few developer releases where we might discover
> and
> sort out possible problems with newer versions of GhostScript on our
> crosscompiled distributions seems sensible.
> 
> Personally, I don't really like the look of this patch at all as it is
> very specifically useful only for one platform, and apart from that
> I'd
> very much want to avoid seeing any wide character functions in
> LilyPond
> code: that's rarely exercised library code of often dubious
> reliability
> or availability, and most programmers are not overly comfortable
> utilizing it.  I'd very much prefer that we'd find a better
> encapsulated
> solution, probably by utilizing the encoding support of GUILEv2.
> Which
> again suggests that a moratorium on this patch makes sense, but that
> one
> would not be "until David gets encodings sorted out so that we can
> compile and test on GUILEv2" but rather on "until we are able to ditch
> GUILEv1 altogether".  And given the current enthusiasm for working on
> the GUILEv2 migration and the likelihood of necessary bug fixes in
> GUILE
> itself becoming available as we discover problems, the latter is
> likely
> quite longer than a month away.
> 
> The problem this patch tries to address is not a recent problem, and
> the
> workaround ("don't use special Unicode characters in filenames") is
> obvious.  So the urgency seems limited.  I very much agree that
> LilyPond
> should not balk at file and directory names containing accented
> characters on all platforms.  But I think we will be doing ourselves
> our
> favor to postpone addressing this problem until we have moved LilyPond
> to GUILEv2, and then look for the best-matching solution within _this_
> framework.  Otherwise we complicate the coding issue work for GUILEv2
> migration and that's something we can afford less than prolonging the
> current filename encoding situation.
> 
> -- 
> You received this message because you are the owner of the issue.
> You may adjust your notification preferences at:
> https://code.google.com/hosting/settings
> 
> Reply to this email to add a comment or make updates.
> 
> Description:
> Add support for unicode filenames on Windows
> 
> On Windows, LilyPond can't handle
> unicode filenames.
> 
> The patch replaces main(), and
> hooks filename related functions.
> This converts between UTF-16
> unicode (Windows) and UTF-8 unicode
> (LilyPond, libguile etc.).
> 
> LilyPond can handle unicode filenames
> for *.ly, *.mid, *.ps.
> 
> *.pdf is not supported yet.
> Ghostscript-8.70 that is included
> in the binary distribution of LilyPond
> can't handle unicode filenames.
> This requires Ghostscript-9.10 or later
> 
> Please review this at https://codereview.appspot.com/206640043/
> 
> Affected files (+523, -0 lines):
>   A flower/include/mingw-utf8.hh
>   A flower/include/mingw-utf8-conv.hh
>   A flower/include/mingw-utf8-func.hh
>   A flower/include/mingw-utf8-hook.hh
>   A flower/mingw-utf8-conv.cc
>   A flower/mingw-utf8-func.cc
>   A flower/mingw-utf8-hook.cc
>   A flower/mingw-utf8-main.cc
>   M lily/GNUmakefile
>   M lily/main.cc