Re: How to make GNU Guile more successful

[Thomas Morley]

Hi Arne,

as promised here my thoughts/findings.
All tested with a LilyPond build from the rebased branch
dev/guile-v2-work, with some additional patches on top of it:

diff --git a/lily/general-scheme.cc b/lily/general-scheme.cc
index 1168ee9..db376d11 100644
--- a/lily/general-scheme.cc
+++ b/lily/general-scheme.cc
@@ -267,6 +267,7 @@ LY_DEFINE (ly_dimension_p, "ly:dimension?", 1, 0,
0, (SCM d),
 /*
   Debugging mem leaks:
 */
+/*
 LY_DEFINE (ly_protects, "ly:protects",
            0, 0, 0, (),
            "Return hash of protected objects.")
@@ -278,6 +279,7 @@ LY_DEFINE (ly_protects, "ly:protects",
   return programming_error ("ly:protects is not supported in Guile 2.1");
 #endif
 }
+*/

 LY_DEFINE (ly_gettext, "ly:gettext",
            1, 0, 0, (SCM original),
diff --git a/scm/lily.scm b/scm/lily.scm
index d3164e4..23b1647 100644
--- a/scm/lily.scm
+++ b/scm/lily.scm
@@ -836,10 +836,11 @@ messages into errors.")

 (define-public (dump-gc-protects)
   (set! gc-protect-stat-count (1+ gc-protect-stat-count))
-  (let* ((protects (sort (hash-table->alist (ly:protects))
-                         (lambda (a b)
-                           (< (object-address (car a))
-                              (object-address (car b))))))
+  (let* (;(protects (sort (hash-table->alist (ly:protects))
+         ;                (lambda (a b)
+         ;                  (< (object-address (car a))
+         ;                     (object-address (car b))))))
+         (protects '())
          (out-file-name (string-append
                          "gcstat-" (number->string gc-protect-stat-count)
                          ".scm"))


diff --git a/ly/scheme-sandbox.ly b/ly/scheme-sandbox.ly
index 7dec0dc..1f233c0 100644
--- a/ly/scheme-sandbox.ly
+++ b/ly/scheme-sandbox.ly
@@ -13,4 +13,8 @@
 % requirements may be different.

 #(newline)
-#(scm-style-repl)
+#(if (guile-v2)
+     (begin
+       (use-modules (system repl repl))
+       (start-repl))
+     (scm-style-repl))



The ly-file used for testing can be found at
http://lists.gnu.org/archive/html/lilypond-user/2016-11/msg00948.html
As the author says, this file was not finished, so several errors and
warnings because of suboptimal user-input happen.
Quite typical for ongoing work and good for testings.
Compiling the file Urtext.ly returns a 159-pages-pdf.

2017-03-05 3:05 GMT+01:00 Thomas Morley <address@hidden>:
> 2017-03-05 0:44 GMT+01:00 Arne Babenhauserheide <address@hidden>:

>> I worked on testing Lilypond with user installed Guile 2.x, does that
>> count?
>>
>> Just to have it recorded somewhere, here’s a patch to lilypond along
>> with a copy of the bash history of the setup (cleaned up, it was many
>> times as long):
>>
>> ## patch
>>
>> From bd2ffea6f4c4c1ede13f5ac82d0a8ce31ccfe3c7 Mon Sep 17 00:00:00 2001
>> Subject: [PATCH] Build fixes for Guile 2.1.x (not yet functional)
>>
>> ---
>>  configure.ac         | 7 ++++++-
>>  lily/pdf-scheme.cc   | 4 ++++
>>  scm/memory-trace.scm | 3 ++-
>>  3 files changed, 12 insertions(+), 2 deletions(-)
>>
>> diff --git a/configure.ac b/configure.ac
>> index d77ea15..393976b 100644
>> --- a/configure.ac
>> +++ b/configure.ac
>> @@ -267,7 +267,12 @@ STEPMAKE_FREETYPE2(freetype2, REQUIRED, 2.1.10)
>>  STEPMAKE_WINDOWS
>>
>>  # guile executable for some scripts
>> -STEPMAKE_GUILE(OPTIONAL, 1.8.2, 1.9.0)
>> +if test "$GUILEv2" = "yes"
>> +then
>> +    STEPMAKE_GUILE(OPTIONAL, 2.0.7, 2.2.0)
>> +else
>> +    STEPMAKE_GUILE(OPTIONAL, 1.8.2, 1.9.0)
>> +fi

The above is already in, see
$ git log -p 91ff9563ebe1c1cd720ad1a44890e7375fd83da8


>>  # perl for help2man and for mf2pt1.pl
>>  STEPMAKE_PERL(REQUIRED)
>> diff --git a/lily/pdf-scheme.cc b/lily/pdf-scheme.cc
>> index da2ce2c..f5ae70c 100644
>> --- a/lily/pdf-scheme.cc
>> +++ b/lily/pdf-scheme.cc
>> @@ -91,7 +91,11 @@ LY_DEFINE (ly_encode_string_for_pdf, 
>> "ly:encode-string-for-pdf",
>>     *                  (string->utf16 str 'big)
>>     */
>>    if (g)
>> +#if GUILEV2
>> +    return scm_take_locale_stringn (g, bytes_written); // scm_take_str 
>> eventually frees g!
>> +#else
>>      return scm_take_str (g, bytes_written); // scm_take_str eventually 
>> frees g!
>> +#endif
>>    else
>>      return str;
>>  }

Instead of the above, currently this code is in:

diff --git a/lily/pdf-scheme.cc b/lily/pdf-scheme.cc
index da2ce2c..61cf382 100644
--- a/lily/pdf-scheme.cc
+++ b/lily/pdf-scheme.cc
@@ -84,14 +84,16 @@ LY_DEFINE (ly_encode_string_for_pdf,
"ly:encode-string-for-pdf",
   free (p);

   /* Convert back to SCM object and return it */
-  /* FIXME guile-2.0: With guile 2.0 the internal representation of a string
-   *                  has changed (char vector rather than binary bytes in
-   *                  UTF-8). However, with guile 2.0, ly:encode-string-for-pdf
-   *                  is no longer needed and can be replaced by the new
-   *                  (string->utf16 str 'big)
-   */
   if (g)
-    return scm_take_str (g, bytes_written); // scm_take_str eventually frees g!
+    {
+      /*
+       * Return the raw byte representation of the UTF-16BE encoded string,
+       * in a locale independent way.
+       */
+      SCM string = scm_from_latin1_stringn (g, bytes_written);
+      free(g);
+      return string;
+    }
   else
     return str;
 }
See:
git log -p d15c38c0ddd4c04edcf82cda50ca30f6dc4941fa

Though, I tested your suggestion as well.
It changes one of our regtests, utf-8.ly. But I have to admit I tested
only this one (not all regtests) and did not try to build the docs.
(All this stuff is pretty heavy for my admittedly weak laptop, it
lasts ages...)
Compiling the above mentioned Urtext.ly-file with your proposal shows
no significant difference for timings (see below)


>> diff --git a/scm/memory-trace.scm b/scm/memory-trace.scm
>> index d8ffeb9..9ebd722 100644
>> --- a/scm/memory-trace.scm
>> +++ b/scm/memory-trace.scm
>> @@ -2,7 +2,8 @@
>>
>>  (define-module (scm memory-trace))
>>  (use-modules (lily)
>> -             (ice-9 format))
>> +             (ice-9 format)
>> +             (ice-9 threads))
>>
>>  (define-public (mtrace:start-trace freq)
>>    (set! usecond-interval (inexact->exact (/ 1000000 freq)))
>> --
>> 2.10.2

Thanks for the above.
It fixes an annoying warning when LilyPond uses guile2.
I'll have to test, but I guess it's compatible with guile-1.
If so I'll likely check it in, ofcourse giving proper credits ;)

[...]
>>
>> PS: If this provides at least a small step towards guile 2 in lilypond,
>>     it’s worth its while.
>> --
>> Unpolitisch sein
>> heißt politisch sein
>> ohne es zu merken
>
> Hi Arne,
>
> many thanks for your work!
> I'll have a closer look tomorrow (it's in the middle of the night here)...
>
> The main problem is not that we can't build lilypond with 2.0.14 or 2.1.7
> Checkout the branch dev/guile-v2-work in the lilypond-repository (you
> will need to rebase it against current master) and you'll be able to
> build LilyPond with guile-2.0.13, so I'm pretty confident it will work
> with 2.0.14. For building with guile 2.1.7 an additional patch is
> needed, currently I've simply deleted all about ly:protects
> (brute-force, I know...).
>
> Though the perfomance of LilyPond is dramatically slowed down. Making
> it effectively unusable for any longer input.
> I'm currently testing this for builds with guile 1.8.8, 2.0.14 and
> 2.1.7 and will report tomorrow.

Here some timing values

(1)
lilypond-2.19.52 using guile 1.8.7
(I would have prefered to build lilypond with a guile-1.8.8 build from
the guile-repository. Though my try to build it from the
branch_release-1-8 failed. Instead attempting to fix it, I then used a
released lilypond-version)

real    8m16.191s
user    6m39.864s
sys    0m10.860s

(2)
guile-2.0.14 build from guile-git-repository, branch remotes/origin/stable-2.0
lilypond-2.19.56, build from local branch dev/guile-v2.2-work

real    34m11.762s
user    45m11.316s
sys    0m5.604s

(3)
guile-2.1.7 build from guile-git-repository, branch master
(I've got this warning:
configure: WARNING: *** GNU Readline is too old on your system.
configure: WARNING: *** You need readline version 2.1 or later.
No idea whether this may have an impact on lilyponds compiling-time
I'll have to test.)
lilypond-2.19.56, build from local branch dev/guile-v2.2-work

real    67m29.132s
user    93m14.812s
sys    0m7.332s


Thanks,
  Harm