Automated testing for users' LilyPond collections with new development versions

[Karlin High]

This message intrigued me:

<https://lists.gnu.org/archive/html/lilypond-devel/2022-11/msg00222.html>

In it, Eric Benson reported a setup that allows testing new versions of 
LilyPond on a sizable body of work in a somewhat automated fashion.

Now, could automation like that also make use of the infrastructure for 
LilyPond's regression tests?

<http://lilypond.org/doc/v2.23/Documentation/contributor/regtest-comparison>

What effort/value would there be in making an enhanced convert-ly tool 
that tests a new version of LilyPond on a user's entire collection of 
work, reporting differences between old and new versions in performance 
and output?

Enabling something like this:

* New release of LilyPond comes out. Please test.

* Advanced users with large collections of LilyPond files do the 
equivalent of "make test-baseline," but for their collection instead of 
LilyPond's regtests. Elapsed time is recorded, also CPU and RAM info as 
seems good.

* New LilyPond gets installed

* Upgrade script runs convert-ly on the collection, first offering 
backup via convert-ly options or tarball-style.

* Equivalent of "make check" runs

* A report generates, optionally as email to lilypond-devel, with 
summary of regression test differences and old-vs-new elapsed time.

Ideally, this could quickly produce lots of good testing info for 
development versions of LilyPond, in a way encouraging user participation.
--
Karlin High
Missouri, USA