Project direction with testing changes (branches and patches)

[Christopher Baines]

Hey,

This is sort of a followup to [1], at least I think that's the last main
email I sent out about testing changes (although I didn't use that
term). I did also send out some notes from the Guix Day event back in
February 2021 though [2].

1: https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00583.html
2: https://lists.gnu.org/archive/html/guix-devel/2021-02/msg00125.html

Back in early 2020, I managed to start work on the Guix Build
Coordinator [3]. That was meant to enable running reliable and
performant substitute servers, but also meant to enable the kind of
testing and quality assurance work that I had been thinking about,
mostly through the perspective of testing patches.

3: https://lists.gnu.org/archive/html/guix-devel/2020-04/msg00323.html

Getting the benefits to users didn't go as smoothly as I'd hoped, but
since bordeaux.guix.gnu.org [4] launched back in June, there's a chance
that the work on the Guix Build Coordinator has benefited users of Guix
through improved substitutes.

4: 
https://guix.gnu.org/en/blog/2021/substitutes-now-also-available-from-bordeauxguixgnuorg/

As I said in [1], I did do some work last year to use the Guix Build
Coordinator for testing patches and branches. Unfortunately the setup
I'm using is currently not operating, I was having issues with running
out of disk space on the main server, and I haven't got around to
spending the time/money to resolve that.

I want to get another iteration of the patch testing setup working, but
recent experiences with working on providing substitutes has made me
think that discussing the direction with maintainers and as a project is
almost more important.

So, I think I've recently switched to thinking about the problem as one
of testing changes, rather than just testing patches. Since both patch
series, and branches are used to propose changes, I think this makes
sense.

In abstract, when testing a change, I would break down the problem as
follows:

  - You need to work out what's affected by the change, so that you can
    assess the impact

  - Once you know what's effected, you can then build those
    packages/system tests/... and compare the build statuses and outputs
    against some baseline

  - Then there's the general UI component, ideally a first time
    contributor would be able to take advantage of automatic feedback
    about a patch they submit. There's multiple other groups of users
    though, like patch reviewers, and committers for example.

I think the first two sub-problems are effectively solved. The Guix Data
Service is able to determine the changes between two revisions (assuming
it's processed them). The Guix Build Coordinator can then be used to
build the relevant packages/system tests, and report that information
back to the Guix Data Service.

The UI part is much less certain, I've done some work with Patchwork,
and I do have some ideas in mind, but there's still more thinking and
work to do in this area.

Before pressing on though, I think it would be good to know if this is a
viable direction?

Currently, there's no automated testing of patches, and testing of
branches is limited to the information that Cuirass provides on failed
builds. What I'm proposing for the future is: using the Guix Data
Service together with the Guix Build Coordinator to analyse the effects
of changes, whether that be from a patch series or a branch. I realise
that I've already been experimenting with this, what I'm mostly
referring to here is moving towards this being the documented approach,
maintained by the project, not just me.

So yes, is this something that people want, or don't want? If you're
uncertain and have questions, it would be good to know what those
questions are?

Thanks,

Chris

signature.asc
Description: PGP signature