Re: guile-2.0 and debian

[Thomas Morley]

2016-11-17 13:05 GMT+01:00 Antonio Ospite <address@hidden>:
> On Tue, 15 Nov 2016 00:55:52 +0100
> Thomas Morley <address@hidden> wrote:
>
>> Hi Antonio,
>>
>> first of all, thanks a lot for working on this!
>>
>> Regrettable during a working week I have very little time to work on a
>> serious issue like this one.
>>
>
> I usually have some spare time on Wednesdays, so... read below.
>
>> That said:
>>
>> 2016-11-14 13:13 GMT+01:00 Antonio Ospite <address@hidden>:
>> > Hi everyone,
>> >
>> > On Mon, 14 Nov 2016 01:15:52 +0100
>> > David Kastrup <address@hidden> wrote:
>> >
>> >> Thomas Morley <address@hidden> writes:
>> >>
>> >> > Short update:
>> >> > I managed to get guile 2.0.13 and guile-2.0-dev for 2.0.13 (grabed it
>> >> > from Zesty Zapus which will be Ubuntu 17.04)
>> >> > Successful make with all of Antonios patches (didn't try make doc so 
>> >> > far).
>> >> >
>> >
>> > In the latest version of my notes I forgot to mention that I am using
>> > 2.0.13 from Debian unstable. It was in the first draft.
>> >
>> >> > Though, _every_ compilation of a .ly will cause a gs-error.
>> >> >
>> >> > Even for { c1 } I get:
>> >> >
>> >> > lilypond-git atest-40.ly
>> >> > GNU LilyPond 2.19.51
>> >> > Processing `atest-40.ly'
>> >> > Parsing...
>> >> > Interpreting music...
>> >> > Preprocessing graphical objects...
>> >> > Finding the ideal number of pages...
>> >> > Fitting music on 1 page...
>> >> > Drawing systems...
>> >> > Layout output to `/tmp/lilypond-6s1XBB'...
>> >> > Converting to `atest-40.pdf'...
>> >> > warning: `(gs -q -dSAFER -dDEVICEWIDTHPOINTS=595,28
>> >> > -dDEVICEHEIGHTPOINTS=841,89 -dCompatibilityLevel=1.4 -dNOPAUSE -dBATCH
>> >> > -r1200 -sDEVICE=pdfwrite -sOutputFile=atest-40.pdf -c.setpdfwrite
>> >> > -f/tmp/lilypond-6s1XBB)' failed (256)
>> >> >
>> >> > fatal error: failed files: "atest-40.ly"
>> >> >
>> >> > Will try to continue tomorrow, too late today.
>> >>
>> >> That one is going to be another (re)encoding error, this time when
>> >> writing PostScript.  It may or may not be a different problem than the
>> >> ones I experienced: Masamichi-san fairly recently made some changes to
>> >> font handling that also were concerned with byte transparency.
>>
>
> I compared the postscript output of the same lilypond git revision built
> with both guile-1.8 (in a debian stable container) and guile-2.0, and by
> looking closely at this issue the conversion from postscript to pdf
> fails for two possible reasons:
>
>   1. The numbers are represented with the decimal separator from the
>      locale instead of a period;
>
>   2. The embedded fonts are encoded in UTF-8, while it looks like that
>      when using guile-1.8 the postscript file is encoded in latin1:
>
>
> I first tried this awful workaround:
>
> ---------------------------------------------------------------------
> diff --git a/scm/framework-ps.scm b/scm/framework-ps.scm
> index a404119..fcbea05 100644
> --- a/scm/framework-ps.scm
> +++ b/scm/framework-ps.scm
> @@ -28,6 +28,8 @@
>               (scm clip-region)
>               (lily))
>
> +(setlocale LC_ALL "C")
> +
>  (define format ergonomic-simple-format)
>
>  (define framework-ps-module (current-module))
> ---------------------------------------------------------------------
>
> It worked, but only for the first processed file, so I came up with
> a somewhat better one:
>
> ---------------------------------------------------------------------
> diff --git a/scm/framework-ps.scm b/scm/framework-ps.scm
> index a404119..23de031 100644
> --- a/scm/framework-ps.scm
> +++ b/scm/framework-ps.scm
> @@ -583,6 +583,9 @@ mark {ly~a_stream} /CLOSE pdfmark
>           (page-number (1- (ly:output-def-lookup paper 'first-page-number)))
>           (page-count (length page-stencils))
>           (port (ly:outputter-port outputter)))
> +
> +    (set-port-encoding! port "ISO-8859-1")
> +    (setlocale LC_NUMERIC "C")
>      (initialize-font-embedding)
>      (if (ly:get-option 'clip-systems)
>          (clip-system-EPSes basename book))
> @@ -601,6 +604,7 @@ mark {ly~a_stream} /CLOSE pdfmark
>       page-stencils)
>      (display "%%Trailer\n%%EOF\n" port)
>      (ly:outputter-close outputter)
> +    (setlocale LC_NUMERIC "")
>      (postprocess-output book framework-ps-module (ly:output-formats)
>                          basename tmp-name #f)))
> ---------------------------------------------------------------------
>
> With this is in place overriding the LANG from the command line to
> overcome the ghostscript failure is not needed anymore, and that's good
> because it turned out to be the cause of the other issues, more below.
>
> [...]
>> You already wrote about stuff like
>>
>> bääh = { c1 }
>> \new Staff \bääh
>>
>> not working and suggested "A workaround is to only use ASCII
>> characters in macro names"
>>
>
> This issue goes away if you stop overriding the locale on the command
> line, that workaround fixed the postscript output but was interfering
> with the file parsing.
>
>> I know several users who would need to fix/rewrite most of their code
>> if we would go for this. But it would be considerable if really,
>> really nothing else is possible, imho
>>
>
> Luckily that will not be necessary, and I think my patch 0008 can be
> dropped as well.
>
>> But this one is the current showstopper:
>>
>> { c1^\markup "büüh" }
>>
>> See attached png.
>> Making LilyPond unusable for everyone.
>>
>>
>> Can you confirm the image?
>>
>
> Yes, I too was getting this, IIUC the "??" are put in by
> scm_to_locale_string(), the doc
> (https://www.gnu.org/software/guile/manual/html_node/Conversion-to_002ffrom-C.html#Conversion-to_002ffrom-C)
> says:
>
>    If it [the conversion strategy] is substitute, a replacement
>    character, such as a question mark, will be inserted in its place.
>
> That too is caused by overriding the locale on the command line: the
> characters in that string cannot be represented in the "C" locale.
>
> Since overriding the locale on the command line is not necessary anymore
> for the ghostscript problem, and it breaks parsing the input files,
> let's not do it.
>
> However, as soon as I started using the native locale (it_IT.utf8)
> I got some warnings:
>
> ---------------------------------------------------------------------
> (process:18706): Pango-WARNING **: Invalid UTF-8 string passed to 
> pango_layout_set_text()
> ---------------------------------------------------------------------
>
> and in the final files only a part of the "büüh" string was rendered,
> however the "ü" was rendered correctly.
>
> So I added a printout to see what was going on:
>
> ---------------------------------------------------------------------
> diff --git a/lily/lily-guile.cc b/lily/lily-guile.cc
> index 2c519ec..9c0c10c 100644
> --- a/lily/lily-guile.cc
> +++ b/lily/lily-guile.cc
> @@ -132,6 +132,7 @@ ly_scm2string (SCM str)
>        result.resize (len);
>        scm_to_locale_stringbuf (str, &result.at (0), len);
>      }
> +  fprintf(stderr, "%s: len: %d result: '%s'\n", __func__, len, 
> result.c_str());
>    return result;
>  }
> ---------------------------------------------------------------------
>
> with guile-1.8:
> ---------------------------------------------------------------------
> ly_scm2string: len: 6 result: 'büüh'
> ---------------------------------------------------------------------
>
> with guile-2.0:
> ---------------------------------------------------------------------
> ly_scm2string: len: 4 result: 'bü�'
>
> (process:18706): Pango-WARNING **: Invalid UTF-8 string passed to 
> pango_layout_set_text()
> ---------------------------------------------------------------------
>
> In ly_scm2string() I see that scm_c_string_length() is used, by looking
> at the documentation
> (https://www.gnu.org/software/guile/manual/html_node/String-Selection.html#String-Selection)
> I read:
>
>         Return the number of characters in string.
>
> So 4 characters looks correct to me, even if they take 6 bytes.
>
> IMHO it can be safer not to mix scm_c_string_length() and
> scm_to_locale_stringbuf().
>
> A possible fix might be:
>
> ---------------------------------------------------------------------
> diff --git a/lily/lily-guile.cc b/lily/lily-guile.cc
> index 2c519ec..7df6a42 100644
> --- a/lily/lily-guile.cc
> +++ b/lily/lily-guile.cc
> @@ -126,12 +126,13 @@ ly_scm2string (SCM str)
>  {
>    assert (scm_is_string (str));
>    string result;
> -  size_t len = scm_c_string_length (str);
> +  size_t len;
> +  char *c_string = scm_to_locale_stringn (str, &len);
>    if (len)
>      {
> -      result.resize (len);
> -      scm_to_locale_stringbuf (str, &result.at (0), len);
> +      result.assign (c_string, len);
>      }
> +  free (c_string);
>    return result;
>  }
>
> ---------------------------------------------------------------------
>
> The code above also works with guile-1.8.
>
> With the change from above and the encoding/locale workaround in
> scm/framework-ps.scm I seem to get an acceptable visual result with your
> examples using either guile-1.8 or guile-2.0.
>
> The output does not match exactly between the two versions, the spacing
> looks a little different, but maybe we are approaching an "usable",
> albeit still experimental, state?
>
> Ciao ciao,
>    Antonio
>
> --
> Antonio Ospite
> https://ao2.it
> https://twitter.com/ao2it
>
> A: Because it messes up the order in which people normally read text.
>    See http://en.wikipedia.org/wiki/Posting_style
> Q: Why is top-posting such a bad thing?

Hi Antonio,

as said, no time to dive in deeper, though here some observations (my
test-file attached.)

- toplevel-markups with special characters working
- ly-identifier with special characters working
- context-names with special characters and assigned Lyrics working

- scheme/guile-identifier with special characters _not_ working
- pdf-meta-data with special characters _not_ working:

exiftool atest-40.pdf
ExifTool Version Number         : 10.10
File Name                       : atest-40.pdf
...skipping...
Title                           : ??b??h
Creator                         : LilyPond 2.19.51

Last, but _most_ serious issue:

make LANGS='' doc
always fails in the regtests with

(1) gs-warnings like
warning: `(gs -q -dNOSAFER -dEPSCrop -dCompatibilityLevel=1.4
-dNOPAUSE -dBATCH -r1200 -sDEVICE=pdfwrite
-sOutputFile=./11/lily-41c82a1e.pdf -c.setpdfwrite
-f./11/lily-41c82a1e.eps)' failed (256)
although they do not happen, if I compile the relevant file separately.

(2) and finally aborts with:
Processing `./0b/lily-d193a1c3.ly'
Parsing...
Renaming input to:
`/home/hermann/lilypond-git/input/regression/markup-cyclic-reference.ly'lilypond:
dynwind.c:121: scm_dynwind_end: Assertion `WINDER_P (entry)' failed.
Aborted (core dumped)

The last one, probably because the two patches about it, one provided
by Antonio and the one David checked in, have
scm_dynwind_end ();
at different places.
Likely I should have applied only one of them

Will try to sort this out - too late today, though.


Sorry to be of not more help,
  Harm

atest-40.ly
Description: Text Data