Re: OMG - performance issue with Scheme engraver

[David Kastrup]

Urs Liska <address@hidden> writes:

> after completing most of the work of reviewing the scholarly.annotate
> module I realize that it (presumably one small change) is a total
> performance killer, and I need some help tracking it down.

Quadratic behavior or worse.

> Observations:
>
>  * The obvious stage of slow-down is when the log prints Interpreting
>    music...[8][16][24] etc.

Does it get slower over time?

> Depending on the number of contexts I have consisted the engraver to
> the first "Finalize" may be printed *before* the [8], so I'm not
> really sure how reliable the order of log output really is here. I'm
> also wondering why "process-acknowledged" is printed before
> "acknowledger" ...

process-acknowledged is called for any grob/spanner, not just the ones
your engraver is listening to.  And currently it is called far more
often than I consider nice or desirable (in particular, it is called in
lower contexts before the grob may even have been acknowledged in higher
contexts) but my attempts to fix that have so far crashed in the
vertical alignment code.

So currently you just have the guarantee that process-acknowledged will
run after a grob has been announced in all contexts, but it may run
before that as well.

> How could I proceed to further track down the issue? But maybe my own
> suspicions already help bringing someone on the right track?

Quadratic behavior or worse.

> I see two changes to the previous code that I think could have such
> impact, one more and one less.

> 2) The engraver works upon grobs that have a certain grob-property
> attached (an 'input-annotation alist that is attached in the
> music-function).
>
> In the earlier code that annotation was attached through \once
> \override, the new code uses (propertyTweak).
>
> In the earlier code I did that test for the property in the
> (acknowledgers) clause and simply skipped all irrelevant grobs. All
> grobs with an annotation were collected in a list and that list later
> processed in (finalize).
>
> In a recent discussion
> (http://lists.gnu.org/archive/html/lilypond-user/2018-06/msg00218.html)
> I was directed to move that test to (process-acknowledged) because a)
> the tweaks weren't available as the grob property earlier and b) this
> was the place to do that.
>
> In order to achieve that I had to use (acknowledgers) to add *all*
> grobs to a list and iterate over that list in (process-acknowledged).

What does "add" mean?

> I have the impression that *this* is the performance problem: creating
> a list with references to *all* grobs in the score, regardless of
> whether they are needed at all. Which would explain why the problem
> exists even if not a single annotation is present in the score.

Creating such a list is not a performance but a memory problem unless
you do something stupid during the creation that will access the whole
list every time you add to it.

Like making sure you don't add the same element multiple times.  If you
want to do that, do it in a private list in process-acknowledged (which
is called too often) and then move the cleaned list to the front(!) of
your big list at stop-translation-timestep.  That limits potentially
quadratic behavior (or worse) to small lists.

-- 
David Kastrup