Re: Learning Lilypond internals

[Thomas Morley]

Am Fr., 12. Aug. 2022 um 07:45 Uhr schrieb Andrew Bernard
<andrew.bernard@mailbox.org>:
>
> Although I have long experience in software development, the Lilypond source 
> is a body of work that I have never really been able to comprehend. This is a 
> pity as I always wanted to help with development. Now, when I have technical 
> issues need knowledge of internals, such as my recent effort relating to 
> noteheads, there is a handful of deeply learned people who come to my 
> assistance, and I am grateful. What I want to ask today is this. The 
> solutions seem to pull commands out of thin air with deep understanding, and 
> I have no idea how to find out this stuff for my self. The internals 
> reference manual is hardly discursive in nature. So Jean, Valentin, Harm 
> (still here?), how have you come to acquire this deep knowledge? I don't know 
> how or where to start.

Yeah, I'm still here :)
Though, my real life job became more time consuming, especially during
Covid-lockdowns my workload exploaded.
Additionally I'm getting older, loosing energy and motivation to work
on every problem which arrives. Thus I'm a little less active than I
used to be.

My own history with LilyPond is similar to Jean's in some regards,
different in others.

First, I didn't had any programming experience before using LilyPond.
No python, no guile, no C++.

Being a guitarist, I soon felt the need to extend LilyPond to fit my
needs. Notation of classical guitar is one of the most complex ones:
Always one Staff with multiple Voices, lots of fingering instructions
(Fingering, StrokeFinger, StringNumber, barre, flageolets, etc). Btw,
we still don't have a barre-sign which fits everyones needs.
Thus I started to learn scheme/guile.
At that time only our Extending Manual and several online
scheme-tutorials were available.

Our Extending Manual is not that bad. It has a short scheme-tutorial.
And demonstrates several basic codings for markup-(list-)commands,
music-functions and "difficult tweaks". At the time I started
scheme/event-functions were not invented, unpure-pure-containers
neither, afair.
So I started to do some homework: defining lots of custom
markup-commands, markup-list-commands and music-functions.
It made me familar with basic coding principles in scheme/lilypond.
I joined the german lilypond forum to get help where my so far
accumulated knowledge ended.
Soon I joined this mailing-list to get even more help.
In return to that help I tried to answer questions from other users.
For some time I actually looked at every user-request, at least
compiling the problem and posting an answer, if I found one.
This increased my knowledge enormously. I think, working on
user-requests on the mailing-list can't be over-estimated regarding
one's own insights.
Having done a LSR (ofcourse with help from others) increased it even more.
Looking/analysing/playing-around at/with posted code from other big
coders like (at that time) Mike Solomon, David Nalesnik, David Kastrup
and several others were another big step forward as well.

Soon I wished to get some of my custom solutions into the source, so I
learned how to do so. Dealing with git, regression-tests, the
patch-upload/approving cycle, etc.

Nowadays, all became a little easier. We have Jean's great
https://extending-lilypond.readthedocs.io. And our own docs became
better. For development  we use GitLab.

Nevertheless I still don't know python and C++.
Especially the lack of C++-knowledge[*] sometimes hinders me to get to
real and general working solutions for certain problems, p.e. if I
need to look at several different grobs to set a certain behaviour of
the original overriden grob and the needed-to-know properties of those
other grobs are done as unpure-pure-containers.
Furthermore I quite frequently need to experiment with things because
I am not able to deduce behaviour from the C++ source.
Thus I often need a code-example. Not saying I need to copy, but to
know we have a page-post-process hook does not enable me to use it.
Reading the example in Usage Manual is what I needed to became familar
with it. (Btw, there is a recent coding by Jean using it, posted here
on the list.)

> So much of my modernist work needs internals knowledge, and even though I 
> have built up a large library of extension code in Lilypond and Scheme, I am 
> still all at sea.
>
> Any advice? I know Scheme and C++ very well. It's the functions and what they 
> do that I don't understand enough. Here's an example.
>
> LY_DEFINE (ly_stencil_outline, "ly:stencil-outline",
>            2, 0, 0, (SCM stil, SCM outline),
>            R"(
> Return a stencil with the stencil expression (inking) of stencil @var{stil} 
> but
> with outline and dimensions from stencil @var{outline}.
>
> I can understand what dimensions means but what is the 'outline' of a 
> stencil, and how is it used? Just one example of my profound ignorance.
>
>
> Andrew

As I wrote, I often make experiments. Here I'd start with
\markup
  \overlay {
    \stencil
      #(stencil-with-color (make-filled-box-stencil '(0 . 10) '(0 . 10)) red)
    \whiteout
    \stencil
      #(ly:stencil-outline
         (stencil-with-color (make-filled-box-stencil '(0 . 5) '(0 . 5)) cyan)
         (make-filled-box-stencil '(0 . 8) '(0 . 8)))
  }

Probably continuing by researching for stencil-true-extent.
Alas, skylines are not scheme accessible afaik. Ofcourse
show-horizontal/vertical-skylines will help. as Jean wrote, though
that was not invented a few years back.

As sort of a summary, I'd like to emphasize that the work on user
requests on the mailing-list is the most promising part to get more
insight in LilyPond internals.
Ofcourse reading the source-code is another very important part, even
if (as in my case) resticted to scm-files.
Third thing: understanding other posters coding.

Best,
  Harm

[*]
A short story about my attempt to learn C++

Years ago I felt it would be nice to know C++, thus I took some online
tutorial and started.
First there were general explanations and an introduction, then the
chapter about a first program, hello-world.cc, started:
Do this ...
Do that ...
Ok, you're ready.
Call the program!

???
How???
???

It was _not_ explained _how_ to call the program.
Actually I had to ask an experienced programmer.

This made me feel stupid and dump.
In the end I felt really pissed.
If that state is reached, it will last for years, actually more than
decade now, iirc.

The good advice "take another tutorial" _is_ good, but I will not follow it.
Ask again in another decade or better two.