Re: Learning Lilypond internals

[Jean Abou Samra]

Le 12/08/2022 à 07:43, Andrew Bernard a écrit :
> Although I have long experience in software development, the Lilypond 
> source is a body of work that I have never really been able to 
> comprehend. This is a pity as I always wanted to help with 
> development. Now, when I have technical issues need knowledge of 
> internals, such as my recent effort relating to noteheads, there is a 
> handful of deeply learned people who come to my assistance, and I am 
> grateful. What I want to ask today is this. The solutions seem to pull 
> commands out of thin air with deep understanding, and I have no idea 
> how to find out this stuff for my self. The internals reference manual 
> is hardly discursive in nature. So Jean, Valentin, Harm (still here?),


Let's not make this an exhaustive list :-) There are
also Lukas, Aaron, Jan-Peter, and more.


> how have you come to acquire this deep knowledge?



My first interaction with the LilyPond community dates back to
end 2018 (I had been using it for two years before). I found
a bug, and reported it on lilypond-user-fr. To do that, I had
to subscribe to the list. Prior to that, I had no idea what
a mailing list was. Subscribing made me receive emails from
other users asking questions, and I realized that sometimes
I could answer these, and that it would be nice to give something
in return for the help I had received. I read replies from
others as well, and learnt from them by example. I became curious
to learn more, and spent some time with Urs' (unfinished) book
about Scheme and LilyPond
(https://scheme-book.ursliska.de/introduction/index.html).
Then at some point I subscribed to lilypond-user as well, and was
nurtured by the numerous code examples posted here, at the time
most notably by Harm and Aaron. I hope that with
https://extending-lilypond.readthedocs.io
I made things easier for people starting on that path now,
but at that time the official extending manual was just as sparse
as it is today, and nothing like the page linked above was
available, so my own learning was primarily by trying to answer
increasingly difficult questions on the mailing list, and dissecting
snippets posted by others.

Then some day I reported a "bug" on the bug-lilypond list, and
it actually wasn't a bug. I complained that the error message
could have been clearer, and David K. told me that instead of
complaining I could propose a different wording, which I did
as a patch. That patch never got in, but it got me started with
the development tools and community. I initially did lots of
cleanups in LilyPond's Python scripts (lilypond-book, convert-ly,
musicxml2ly, etc.) because I was not really that fluent in Scheme
and I had zero knowledge of C++, but I gradually started to
understand more things by reading discussions between developers.
I have also spent absolutely unreasonable amounts of time reading the
archives of the lilypond-devel list. I was made more familiar with
functional programming through university courses where practicals
were in OCaml, which made me able to write Scheme code more fluently.
The successive lockdowns when Covid arrived were a catalyst that
increased my participation in development, and I learnt the rest
by reading the source while modifying it. Once you have a sufficient
understanding of the core concepts to be able to understand most
parts of the source code, learning becomes so easy that it isn't
funny, because if you want information about anything, you can
consult the ultimate complete definitive guide about everything,
the source. When replying on mailing lists, I always post links
to the documentation when feasible, but most of the time I didn't
get to these links by actually searching the documentation, but
by searching the source and then grepping for more specific keywords
in the documentation, which is faster for me. Of course, it did take
me quite some time to get to that point.

The bottom line is that mailing lists are your friend just as much
as the documentation, and not only in the questions that you ask
yourself.


> I don't know how or where to start.
>
> So much of my modernist work needs internals knowledge, and even 
> though I have built up a large library of extension code in Lilypond 
> and Scheme, I am still all at sea.
>
> Any advice? I know Scheme and C++ very well. It's the functions and 
> what they do that I don't understand enough. Here's an example.
>
>     LY_DEFINE (ly_stencil_outline, "ly:stencil-outline",
>                2, 0, 0, (SCM stil, SCM outline),
>                R"(
>     Return a stencil with the stencil expression (inking) of stencil
>     @var{stil} but
>     with outline and dimensions from stencil @var{outline}.
>
> I can understand what dimensions means but what is the 'outline' of a 
> stencil, and how is it used? Just one example of my profound ignorance.


Try

\version "2.23.11"

{
  \tweak Accidental.show-horizontal-skylines ##t
  \tweak Accidental.show-vertical-skylines ##t
  cis'
}

and you should have an idea of what LilyPond calls "skylines".
They are used for accurate spacing, e.g. when a TextScript is
positioned above its note head, that sort of thing.

The call (ly:stencil-outline stil1 stil2) returns a stencil
that gets drawn the same as stil1, but is treated by the
skyline construction process as if it were stil2. The former
happens in lily/stencil-interpret.cc:

void
interpret_stencil_expression (SCM expr, Stencil_sink *sink, Offset o)
{
  [...]
      else if (scm_is_eq (head, ly_symbol2scm ("with-outline")))
        {
          expr = scm_caddr (expr);
        }


And the latter happens in lily/stencil-integral.cc:


static void
interpret_stencil_for_skyline (Lazy_skyline_pair *skyline,
                               Transform const &transform, SCM expr)
{
  [...]
  else if (scm_is_eq (head, ly_symbol2scm ("with-outline")))
    interpret_stencil_for_skyline (skyline, transform, scm_cadr (expr));


Hope that helps,
Jean