Re: guile-2.0 and debian

[Antonio Ospite]

On Tue, 15 Nov 2016 00:55:52 +0100
Thomas Morley <address@hidden> wrote:

> Hi Antonio,
> 
> first of all, thanks a lot for working on this!
> 
> Regrettable during a working week I have very little time to work on a
> serious issue like this one.
>

I usually have some spare time on Wednesdays, so... read below.

> That said:
> 
> 2016-11-14 13:13 GMT+01:00 Antonio Ospite <address@hidden>:
> > Hi everyone,
> >
> > On Mon, 14 Nov 2016 01:15:52 +0100
> > David Kastrup <address@hidden> wrote:
> >
> >> Thomas Morley <address@hidden> writes:
> >>
> >> > Short update:
> >> > I managed to get guile 2.0.13 and guile-2.0-dev for 2.0.13 (grabed it
> >> > from Zesty Zapus which will be Ubuntu 17.04)
> >> > Successful make with all of Antonios patches (didn't try make doc so 
> >> > far).
> >> >
> >
> > In the latest version of my notes I forgot to mention that I am using
> > 2.0.13 from Debian unstable. It was in the first draft.
> >
> >> > Though, _every_ compilation of a .ly will cause a gs-error.
> >> >
> >> > Even for { c1 } I get:
> >> >
> >> > lilypond-git atest-40.ly
> >> > GNU LilyPond 2.19.51
> >> > Processing `atest-40.ly'
> >> > Parsing...
> >> > Interpreting music...
> >> > Preprocessing graphical objects...
> >> > Finding the ideal number of pages...
> >> > Fitting music on 1 page...
> >> > Drawing systems...
> >> > Layout output to `/tmp/lilypond-6s1XBB'...
> >> > Converting to `atest-40.pdf'...
> >> > warning: `(gs -q -dSAFER -dDEVICEWIDTHPOINTS=595,28
> >> > -dDEVICEHEIGHTPOINTS=841,89 -dCompatibilityLevel=1.4 -dNOPAUSE -dBATCH
> >> > -r1200 -sDEVICE=pdfwrite -sOutputFile=atest-40.pdf -c.setpdfwrite
> >> > -f/tmp/lilypond-6s1XBB)' failed (256)
> >> >
> >> > fatal error: failed files: "atest-40.ly"
> >> >
> >> > Will try to continue tomorrow, too late today.
> >>
> >> That one is going to be another (re)encoding error, this time when
> >> writing PostScript.  It may or may not be a different problem than the
> >> ones I experienced: Masamichi-san fairly recently made some changes to
> >> font handling that also were concerned with byte transparency.
>

I compared the postscript output of the same lilypond git revision built
with both guile-1.8 (in a debian stable container) and guile-2.0, and by
looking closely at this issue the conversion from postscript to pdf
fails for two possible reasons:

  1. The numbers are represented with the decimal separator from the
     locale instead of a period;

  2. The embedded fonts are encoded in UTF-8, while it looks like that
     when using guile-1.8 the postscript file is encoded in latin1:


I first tried this awful workaround:

---------------------------------------------------------------------
diff --git a/scm/framework-ps.scm b/scm/framework-ps.scm
index a404119..fcbea05 100644
--- a/scm/framework-ps.scm
+++ b/scm/framework-ps.scm
@@ -28,6 +28,8 @@
              (scm clip-region)
              (lily))

+(setlocale LC_ALL "C")
+
 (define format ergonomic-simple-format)

 (define framework-ps-module (current-module))
---------------------------------------------------------------------

It worked, but only for the first processed file, so I came up with
a somewhat better one:

---------------------------------------------------------------------
diff --git a/scm/framework-ps.scm b/scm/framework-ps.scm
index a404119..23de031 100644
--- a/scm/framework-ps.scm
+++ b/scm/framework-ps.scm
@@ -583,6 +583,9 @@ mark {ly~a_stream} /CLOSE pdfmark
          (page-number (1- (ly:output-def-lookup paper 'first-page-number)))
          (page-count (length page-stencils))
          (port (ly:outputter-port outputter)))
+
+    (set-port-encoding! port "ISO-8859-1")
+    (setlocale LC_NUMERIC "C")
     (initialize-font-embedding)
     (if (ly:get-option 'clip-systems)
         (clip-system-EPSes basename book))
@@ -601,6 +604,7 @@ mark {ly~a_stream} /CLOSE pdfmark
      page-stencils)
     (display "%%Trailer\n%%EOF\n" port)
     (ly:outputter-close outputter)
+    (setlocale LC_NUMERIC "")
     (postprocess-output book framework-ps-module (ly:output-formats)
                         basename tmp-name #f)))
---------------------------------------------------------------------

With this is in place overriding the LANG from the command line to
overcome the ghostscript failure is not needed anymore, and that's good
because it turned out to be the cause of the other issues, more below.

[...]
> You already wrote about stuff like
> 
> bääh = { c1 }
> \new Staff \bääh
> 
> not working and suggested "A workaround is to only use ASCII
> characters in macro names"
>

This issue goes away if you stop overriding the locale on the command
line, that workaround fixed the postscript output but was interfering
with the file parsing.

> I know several users who would need to fix/rewrite most of their code
> if we would go for this. But it would be considerable if really,
> really nothing else is possible, imho
>

Luckily that will not be necessary, and I think my patch 0008 can be
dropped as well.

> But this one is the current showstopper:
> 
> { c1^\markup "büüh" }
> 
> See attached png.
> Making LilyPond unusable for everyone.
> 
> 
> Can you confirm the image?
>

Yes, I too was getting this, IIUC the "??" are put in by
scm_to_locale_string(), the doc
(https://www.gnu.org/software/guile/manual/html_node/Conversion-to_002ffrom-C.html#Conversion-to_002ffrom-C)
says:

   If it [the conversion strategy] is substitute, a replacement
   character, such as a question mark, will be inserted in its place.

That too is caused by overriding the locale on the command line: the
characters in that string cannot be represented in the "C" locale.

Since overriding the locale on the command line is not necessary anymore
for the ghostscript problem, and it breaks parsing the input files,
let's not do it.

However, as soon as I started using the native locale (it_IT.utf8)
I got some warnings:

---------------------------------------------------------------------
(process:18706): Pango-WARNING **: Invalid UTF-8 string passed to 
pango_layout_set_text()
---------------------------------------------------------------------

and in the final files only a part of the "büüh" string was rendered,
however the "ü" was rendered correctly.

So I added a printout to see what was going on:

---------------------------------------------------------------------
diff --git a/lily/lily-guile.cc b/lily/lily-guile.cc
index 2c519ec..9c0c10c 100644
--- a/lily/lily-guile.cc
+++ b/lily/lily-guile.cc
@@ -132,6 +132,7 @@ ly_scm2string (SCM str)
       result.resize (len);
       scm_to_locale_stringbuf (str, &result.at (0), len);
     }
+  fprintf(stderr, "%s: len: %d result: '%s'\n", __func__, len, result.c_str());
   return result;
 }
---------------------------------------------------------------------

with guile-1.8:
---------------------------------------------------------------------
ly_scm2string: len: 6 result: 'büüh'
---------------------------------------------------------------------

with guile-2.0:
---------------------------------------------------------------------
ly_scm2string: len: 4 result: 'bü�'

(process:18706): Pango-WARNING **: Invalid UTF-8 string passed to 
pango_layout_set_text()
---------------------------------------------------------------------

In ly_scm2string() I see that scm_c_string_length() is used, by looking
at the documentation
(https://www.gnu.org/software/guile/manual/html_node/String-Selection.html#String-Selection)
I read:

        Return the number of characters in string.

So 4 characters looks correct to me, even if they take 6 bytes.

IMHO it can be safer not to mix scm_c_string_length() and
scm_to_locale_stringbuf().

A possible fix might be:

---------------------------------------------------------------------
diff --git a/lily/lily-guile.cc b/lily/lily-guile.cc
index 2c519ec..7df6a42 100644
--- a/lily/lily-guile.cc
+++ b/lily/lily-guile.cc
@@ -126,12 +126,13 @@ ly_scm2string (SCM str)
 {
   assert (scm_is_string (str));
   string result;
-  size_t len = scm_c_string_length (str);
+  size_t len;
+  char *c_string = scm_to_locale_stringn (str, &len);
   if (len)
     {
-      result.resize (len);
-      scm_to_locale_stringbuf (str, &result.at (0), len);
+      result.assign (c_string, len);
     }
+  free (c_string);
   return result;
 }

---------------------------------------------------------------------

The code above also works with guile-1.8.

With the change from above and the encoding/locale workaround in
scm/framework-ps.scm I seem to get an acceptable visual result with your
examples using either guile-1.8 or guile-2.0.

The output does not match exactly between the two versions, the spacing
looks a little different, but maybe we are approaching an "usable",
albeit still experimental, state?

Ciao ciao,
   Antonio

-- 
Antonio Ospite
https://ao2.it
https://twitter.com/ao2it

A: Because it messes up the order in which people normally read text.
   See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?