Documentation of Architecture / Design?

[Han-Wen Nienhuys]

address@hidden writes:
> All,
> 
> I've recently gotten involved in the lilypond cause (2 bug fixes submitted, a
> couple of features underway).  My problem is I don't have a firm understanding
> of how everything comes together.  I've read through the Programmer's 
> Reference
> (PR), but it isn't what I'm looking for.
> 
> I don't know the system well enough to answer even half of these questions.  I
> could start a document by filling in what I've learned so far, but my answers
> would have to be checked, and there are lots of things that would need to be
> filled in by others.

That would be most welcome!

I must admit that when I want to know how a program works, I use grep
and emacs and dive into the source code. The comments and the code
itself are usually more revealing than technical documents.

Can you tell me where you started looking and where you lost it?

> Is anybody interested in helping out on this if I get the ball rolling?

Yes! I'll gladly supply answers. I'll start off with your current
batch of questions. I'm going to give very brief, compact answers, but
do not hesitate to point out what you don't understand, and ask more,
so I know where to expand.

> I think what's needed is a document describing the following things:
> 
> - What's a grob, and how is one used?

Graphical object - they are created from within engravers, either as
Spanners (derived class) -slurs, beams- or Items (also a derived
class) -notes, clefs, etc.

There are two other derived classes System (derived from Spanner,
contaning a "line of music") and Paper_column (derived from Item, it
contains all items that happen at the same moment). They are separate
classes because they play a special role in the linebreaking process.

> - What's a smob, and how is one used?

A C(++) object that is encapsulated so it can be used as a Scheme
object.  See GUILE info, "19.3 Defining New Types (Smobs)"

> - When is each C++ class constructed and used (sort of in the PR)
>       Music classes

In the parser.yy see the macro calls MAKE_MUSIC_BY_NAME().

>       Contexts

Constructed during "interpreting" phase.

>       Engravers

Executive branch of Contexts, plugins that create grobs, usually one
engraver per grob type. Created  together with context.

>       Layout Objects

= grobs

>       Grob Interfaces

These are not C++ classes per se. The idea of a Grob interface hasn't
crystallized well. ATM, an interface is a symbol, with a bunch of grob
properties. They are not objects that are created or destroyed.

>       Iterators (hven't found them at all in the PR)
>       (Others ?)

Objects that walk through different music classes, and deliver events
in a synchronized way, so that notes that play together are processed
at the same moment and (as a result) end up on the same horizontal position.

Created during interpreting phase.

BTW, the entry point for interpreting is ly:run-translator
(ly_run_translator on the C++ side)

> - How do the classes interact with each other
>       A UML diagram would be nice)

> - Which properties are accessible from the class types
>       Can you get to Context properties from a Music object?

You can create music object with a Scheme function that reads context
properties (the \applycontext syntax). However, that function is
executed during Interpreting, so you can not really get Context
properties from Music objects, since music objects are not directly
connected to Contexts. That connection is made by the  Music_iterators

>       Can you get to Music properties from a Context object?

Yes, if you are given the music object within a Context
object. Normally, the music objects enter Contexts in synchronized
fashion, and the synchronization is done by Music_iterators.  

> - What is the relationship between C++ classes and Scheme objects?

Smobs are C++ objects in Scheme. Scheme objects (lists, functions) are
manipulated from C++ as well using the GUILE C function interface
(prefixes: gh_ and scm_)

> - How do Scheme procedures get called from C++ functions?

scm_call_0, scm_call_1. Also scm_c_eval_string (), scm_eval ()

> - How do C++ functions get called from Scheme procedures?

Export a C++ function to Scheme with LY_DEFINE.

> - What is the flow of control in the program?

Good question.  Things used to be clear and cut, but we have Scheme
and SMOBs now, which means that interactions do not follow a very
rigid format anymore. See below for an overview, though.

>       Does the parser make Scheme procedure calls or C++ function
> calls?

Both. And the Scheme calls can call C++ and vice versa. It's nested,
with the SCM datatype as lubrication between the interactions

(I think the word "lubrication" describes the process better than the
traditional word "glue")

>       How do the front-end and back-end get started?

Front-end: a file is parsed, the rest follows from that. Specifically,

Parsing leads to a Music + Music_output_def object (see parser.yy,
definition of toplevel_expression )

A Music + Music_output_def object leads to a Global_context object (see
ly_run_translator ())

During interpreting, Global_context + Music leads to a bunch of
Contexts. (see Global_translator::run_iterator_on_me () )

After interpreting, Global_context contains a Score_context (which
contains staves, lyrics etc.) as a child. Score_context::get_output ()
spews a Music_output object (either a Paper_score object for notation
or Performance object for MIDI).

The Music_output object is the entry point for the backend. (see
ly_render_output () )

The main steps of the backend itself are in

* paper-score.cc , Paper_score::process_

* system.cc , System::get_lines()

* The step, where things go from grobs to .ps or .tex, is in
  System::get_line(): each grob delivers a Stencil (a Device
  independent output description), which is interpreted by our
  outputting backends (scm/output-tex.scm and scm/output-ps.scm)
  to produce TeX and PS.

Interactions between grobs and putting things into .tex and .ps files
have gotten a little more complex lately. Jan has implemented
page-breaking, so now the backend also involves Paper_book,
Paper_lines and other things. This area is still heavily in flux, and
perhaps not something you should want to look at.

>       How do the front-end and back-end communicate?

There is no communication from backend to front-end. From front-end to
backend is simply the program flow: music + definitions gives
contexts, contexts yield output, after processing, output is written
to disk.

> - Where is the functionality associated with KEYWORDs?

See my-lily-lexer.cc (keywords, there aren't that many) and ly/*.ly
(most of the other backslashed \words are identifiers)


> What
> Contexts/Properties/Music/etc. are available when they are
> processed?

What do you mean exactly with this question?

See ly/engraver-init.ly for contexts, see scm/define-*.scm for other
objects.

>       \override
>       \stemDown
>       \relative
>       \notes
>       \autochange
>       (etc.)
> 
> - How do you decide if something is a Music, Context, or Grob property?
>       Why is part-combine-status a Music property when it seems (IMHO)
>       to be related to the Staff context?

The Music_iterators and Context communicate through two channels

* Music_iterators can set and read context properties, idem for
  Engravers and Contexts

* Music_iterators can send "synthetic" music events (which aren't in
  the input) to a context. These are caught by Engravers. This is
  mostly a one way communication channel.

part-combine-status is part of such a synthetic event, used by 
Part_combine_iterator to communicate with Part_combine_engraver.

>       I'm adding a property to affect how \autochange works.  It seems to
>       me that it should be a context property, but the Scheme autochange
>       procecure has a Music argument.  Does this mean I should use
>       a Music property?

\autochange is one of these extra strange beasts: it requires
look-ahead to decide when to change staves. This is achieved by
running the interpreting step twice (see scm/part-combiner.scm , at
the bottom), and storing the result of the first step (where to switch
staves) in a Music property.  Since you want to influence that
where-to-switch list, your must affect the code in
make-autochange-music (scm/part-combiner.scm). That code is called
directly from the parser and there are no official "parsing
properties" yet, so there is no generic way to tune \autochange. We
would have to invent something new for this, or add a separate
argument,

    \autochange #around-central-C ..music..

where around-central-C is some function that is called from
make-autochange-music.

> I can (and have) searched the code for this kind of information, but with
> limited (almost non-existent) comments, it's VERY difficult to get a foothold.
> Also, the PR sometimes doesn't add much clarity:
> 
>       ? Function: Context_specced_music_iterator::constructor
>               Construct a Context_specced_music_iterator music iterator
> 
> I would not be surprised if some eager potential developers turned away 
> because
> of these deficiencies.

Yes that is quite possible. Please keep firing away with questions,
and pinpoint where in the source code you get lost.


-- 

 Han-Wen Nienhuys   |   address@hidden   |   http://www.xs4all.nl/~hanwen