Re: Blockers for Guile 2.2

[Jean Abou Samra]

Hi,

Sorry for the late reply, I hoped to have a merge request
ready to implement but life is getting in the way.
Nevertheless, a proof of concept is this:


From 95794324cd4a637c4735447b672a1de91416cc4a Mon Sep 17 00:00:00 2001
From: Jean Abou Samra <jean@abou-samra.fr>
Date: Sun, 20 Feb 2022 17:00:20 +0100
Subject: [PATCH] WIP: allow separate Guile byte-compilation

Benefits: can be integrated into the build system, can byte-compile
all files even if not used, can byte-cross-compile.

TODO: actually integrate with build system, figure out load paths
etc., also compile files not directly loaded in (lily).
---
 scm/lily.scm | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/scm/lily.scm b/scm/lily.scm
index afc84d39d8..58d4fc39e4 100644
--- a/scm/lily.scm
+++ b/scm/lily.scm
@@ -263,6 +263,13 @@ lilypond-book")
                           "Check every property assignment for types.")
     (clip-systems #f
                   "Generate cut-out snippets of a score.")
+    (guile-compile #f
+                   "Run Guile byte-compilation on LilyPond's Scheme
+files.  This is intended for use when compiling or packaging
+LilyPond.")
+    (guile-compile-target #f
+                          "When using -d guile-compile, specifies the
+platform triplet to generate bytecode for.")
     (crop #f
           "Generate additional, possibly tall
 single-page output file(s) with cropped margins.")
@@ -844,6 +851,11 @@ PIDs or the number of the process."

 ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

+(use-modules (system base compile)
+             (system base target))
+
+(define start-module (current-module))
+
 (define-public (lilypond-main files)
   "Entry point for LilyPond."
   (eval-string (ly:command-line-code))
@@ -855,6 +867,24 @@ PIDs or the number of the process."
              (ly:exit 0 #t)))
   (if (ly:get-option 'gui)
       (gui-main files))
+  (if (ly:get-option 'guile-compile)
+      (begin
+        (for-each
+         (lambda (file)
+           (let* ((full-path (string-append "lily/" file))
+                  (file-name (%search-load-path full-path)))
+             (ly:message "Byte-compiling ~a..." file)
+             (with-target (let ((custom-target(ly:get-option 
'guile-compile-target)))
+                            (if custom-target
+                                (symbol->string custom-target)
+                                %host-type))
+               (lambda ()
+                 (compile-file file-name
+                               #:env start-module
+                               #:output-file (string-append 
"/home/jean/repos/lilypond/scm/" file ".go")
+                               )))))
+         init-scheme-files)
+        (ly:exit 0 #t)))
   (if (null? files)
       (begin (ly:usage)
              (ly:exit 2 #t)))
--
2.32.0



This is in a very experimental stage: it has paths hardcoded
for my setup, it doesn't actually load the files, I still
have to figure out how to integrate it in Autohell, etc.
Nevertheless, it's a proof of concept that it is possible,
without restructuring the scm/ directory at all, to get
Guile bytecode in a build system for a given architecture.



Le 19/02/2022 à 23:21, Jonas Hahnfeld a écrit :
> Am Samstag, dem 19.02.2022 um 23:05 +0100 schrieb Jean Abou Samra:
> > Le 19/02/2022 à 22:43, Jonas Hahnfeld a écrit :
> > > Am Samstag, dem 19.02.2022 um 21:34 +0100 schrieb Jean Abou Samra:
> > > > Le 19/02/2022 à 21:00, Jonas Hahnfeld a écrit :
> > > > > I'm firmly convinced that the order must be
> > > > > 1. only test with Guile 2.2 in CI
> > > > > 2. make configure only look for Guile 2.2 by default
> > > > > 3. do a release with Guile 2.2 only
> > > > > 4. unless emergency happens, drop all code related to Guile 1.8 very
> > > > > soon after
> > > > I disagree with 'very soon after' in point 4. Is this code
> > > > really hurting? We don't have so much of it as far as I can
> > > > see.
> > > All "guile-2" in scm/, GUILEV2 in lily/, and a number of configure
> > > checks. It will be significantly easier to restructure all of our
> > > Scheme code without that.
> > I am not sure about that. It's unclear to me whether we're
> > going to have to move stuff around in the end -- see the helpful
> > reply from Olivier Dion this evening on guile-user/guile-devel --,
> The proposed approach is worse than just doing a GUILE_AUTO_COMPILE
> because we need more code to make it happen. More fundamentally, it
> doesn't address the cross-compilation setup.


It does, see above.


> > and if we do, how would Guile 1 interfere with moving code
> > around?
> It requires testing, and there are constructs such as eval-when that
> Guile 1.8 didn't have. Yes, you can work around that, but it will
> require more code and even more time.


(Adressed above.)


> > > Plus, your plan of keeping code for Guile 1.8 doesn't work / make sense
> > > without keeping GUB working. That is far more complicated and prevents
> > > many future changes.
> > Right now, it is in a working state.
> Because I kept it working so far.
>
> > I am like St Thomas and need examples to be convinced by 'many'. Even for
> > Cairo there is a solution in GUB.
> One example ishttps://gitlab.com/lilypond/lilypond/-/issues/5831,
> another is
> https://lists.gnu.org/archive/html/lilypond-devel/2020-08/msg00159.html
> where (I think) we shouldn't install the fonts as part of LilyPond's
> build system. Both of them would require updates to GUB, which I
> honestly don't want to make.
>
> > To be clear, I'm not happy about keeping it around, but
> > I very much think we shouldn't actively drop it until we
> > have a state that resembles what we eventually want to
> > call stable. Because we don't know what problems are going
> > to pop up with shipping bytecode.
> >
> > > > I find it too risky to lock ourselves into Guile 2 in
> > > > a state where a stable release would not be satisfactory
> > > > (see also below). Plus, a number of problems have been popping up
> > > > with the new binaries. It's great that you're fixing them
> > > > at the speed of light (thanks!), but I'm not convinced by
> > > > waiting only shortly after a new release to call it good
> > > > to go. Would it consume excessive CI minutes if we tested
> > > > with both Guile 1 and 2 for some time? If so, I would
> > > > suggest having optional jobs with Guile 1, basically swapping
> > > > the current roles.
> > > Again, this doesn't make sense: If we want to keep Guile 1.8 working,
> > > the jobs must be mandatory. So we're talking at least twice the CI
> > > minutes, in practice much more because frog can only run one job at a
> > > time.
> > Would that be an acceptable load on the CI infrastructure?
> It's not only about load, it's about every merge request taking 3x as
> long to merge.
>
> > If not, I wouldn't find too much of an issue with optional jobs.
> > Right now, we want to keep Guile 2 working, the jobs are
> > optional, and it keeps working thanks to an occasional
> > 'hotfix'. Those fixes don't even need to be made while we are
> > in the state where LilyPond still accepts being configured
> > with Guile 1. If the transition only takes a month or so,
> > it should normally remain possible to roll back to Guile 1,
> > fixing a few things here and there. It is actively dropping
> > code supporting Guile 1 that I object to at this point.
> I don't share your optimism here, neither on "occasional 'hotfix'", "a
> month or so", nor "fixing a few things here and there". I predict
> things will break, and very quickly if you attempt to do things as
> invasive as making byte compilation work.


Honestly, if you are pessimistic about "a month or so" to
get a setup that could work for a stable release, that
doesn't make me confident about making a full switch now ...

As far as I have understood the proposed plan, it involves
putting the master branch in a state that is fundamentally
unsuitable for production in Guile 2 and leave it so
for months, while dropping Guile 1 support, making it
effectively impossible to get something eventually satisfactory
without the success of experimental work that has no
proof-of-concept yet. In an environment run by volunteers
where any person can be "hit by a bus" at any time, I don't
consider this to be a wise plan.

Of course, if it is taken to a working state, the proof-of-concept
above will change the game. I intend to continue investigating,
but feel free to beat me to it (Autoconf aka Autohell scares
me just as much as TeX, which is to say a lot).

Regards,
Jean