Re: New suggestion to make Lily pond speed up on multiple CPU cores

[Jean Abou Samra]

Le 13/07/2022 à 12:58, Arno Waschk a écrit :
> Hi all,
>
> I know the topic itself is not new, and I am aware that lilypond can't
> easily take advantage from multiple CPU cores as the way it "compiles"
> lilypond notation into a final pdf file does not allow for it.
>
> But there could be an exception to be used for it: Slicing a (big, of
> course, we won't need anything for small ones I think) score along its
> pagebreaks, and calculate those (with a "fast mode" option maybe)
> independently in parallel, just joining the results for the pdf, or
> otherwise very lately after all those layout calculations.
>
> Such pagebreaks could be inserted by
>
> a) a user manually as he wishes for those anyway
>
> b) a user manually for speed up during writing and proof reading a score,
> only removing them for the final result as he does wish for the perfect
> lilypond layout when everything else is ready
>
> c) some logic in early parsing stage for automatic speed up when a user
> wishes so and is fine about possibly suboptimal pagebreakimg while
> writing/proofreading etc.
>
> Sorry if something like that was already suggested, but i did not find such.
>
> If something like this sounds feasible I am happy to invest some of my time
> into development, although I would certainly need some help and guidance
> into lilypond's codebase.



This idea is not entirely new :-) See the end of
https://lilypond.org/google-summer-of-code.html

I don't really know if it's feasible. I see several problems:

1. The pages (or systems) are not completely independent,
   as you can see from

\version "2.23.11"

\paper {
  ragged-right = ##t
}

{
  c'1\<
  \break
  c'1\!
}

{
  c'1\<
  \break
  c'2 2\!
}


Observe the height of the hairpin of the first system.

So you need a system of locks to prevent issues with that.

2. The part of LilyPond's processing that happens on
   broken systems is actually quite small. Most expensive
   things are already done before line breaking. So
   you'd likely need to adapt some code to also work after
   line breaking in order to get a significant speedup.

You may also want to take into consideration:

3. Guile's threading support relies on POSIX threads, so it
   doesn't work on Windows.


I can't comment much more, as I don't know much about
concurrency. Maybe posting on lilypond-devel would attract
more attention (as it's a very general discussion).

Jean