OMG - performance issue with Scheme engraver

[Urs Liska]

Hi all,

after completing most of the work of reviewing the scholarly.annotate module I realize that it (presumably one small change) is a total performance killer, and I need some help tracking it down.

There are a few things I have changed (had to), and one in particular seemed suspicious to me right away.

The situation is that a three-page score that takes ~ 2.1 seconds without the engraver and with the old state of the engraver now needs 5.8 seconds. And a >20 page score that I recall using around 20-25 seconds now needs 57 - so that's obviously nothing to ignore.

Observations:

• The obvious stage of slow-down is when the log prints Interpreting music...[8][16][24] etc.

To monitor more closely I have inserted (ly:message) commands

• "Engraver instantiated" in the closure around the engraver's lambda _expression_
• "Engraver called" between the (lambda (context)) and (make-engraver ...
• "First acknowledger" in the (acknowledgers) clause (prints only once)
• "First process-acknowledged" in the (process-acknowledged) clause (prints only once)

Now the log output prints:

Parsing...
Engraver instantiated
Interpreting music...
Engraver called
Engraver called
Engraver called
First process-acknowledged
First acknowledger[8][16][24]
Finalize
Finalize
Finalize
Preprocessing graphical objects...

where the bar counter proceeds extremely slowly (about 1 second per entry).

Depending on the number of contexts I have consisted the engraver to the first "Finalize" may be printed *before* the [8], so I'm not really sure how reliable the order of log output really is here. I'm also wondering why "process-acknowledged" is printed before "acknowledger" ...

How could I proceed to further track down the issue? But maybe my own suspicions already help bringing someone on the right track?

I see two changes to the previous code that I think could have such impact, one more and one less.

1) In the music-function (i.e. not in the engraver) I determine an "anchor" if it's sequential music or a chord. In these cases I add a reference to that anchor as a music-property to the music _expression_. This is because the music _expression_ is chained through several stages, and I need to keep a reference to that anchor.

This means that the music _expression_ sees the anchor element twice, but I *think* this is only a reference so there should be no performance penalty connected, right?

And I find it very unlikely that this is the issue because the performance problem is triggered by merely \consist-ing the engraver, even if it isn't used at all.

2) The engraver works upon grobs that have a certain grob-property attached (an 'input-annotation alist that is attached in the music-function).

In the earlier code that annotation was attached through \once \override, the new code uses (propertyTweak).

In the earlier code I did that test for the property in the (acknowledgers) clause and simply skipped all irrelevant grobs. All grobs with an annotation were collected in a list and that list later processed in (finalize).

In a recent discussion (http://lists.gnu.org/archive/html/lilypond-user/2018-06/msg00218.html) I was directed to move that test to (process-acknowledged) because a) the tweaks weren't available as the grob property earlier and b) this was the place to do that.

In order to achieve that I had to use (acknowledgers) to add *all* grobs to a list and iterate over that list in (process-acknowledged).

I have the impression that *this* is the performance problem: creating a list with references to *all* grobs in the score, regardless of whether they are needed at all. Which would explain why the problem exists even if not a single annotation is present in the score.

• Does that make sense?
• How can I test this further?
• Is there a way to get rid of this list-of-all-grobs (I don't want to test what that would do to a 100-page score ...) while still catching the grob properties attached through a \tweak?

The code can be seen here: https://github.com/openlilylib/scholarly/blob/editorial-markup-choice-and-annotate/annotate/engraver.ily or by cloning the https://github.com/openlilylib/scholarly repository and checking out the editorial-markup-choice-and-annotate branch (files in annotate/...)-

Any help would be greatly appreciated

Urs