Re: testing out Docker CI scripts?

[Han-Wen Nienhuys]

On Sat, Feb 22, 2020 at 10:01 PM David Kastrup <address@hidden> wrote:
> >> That's oversimplifying the staging->master push process a bit which goes
> >> to considerable pain to make sure that its own copy of staging is still
> >> part of the upstream staging branch before pushing the tested result.
> >> That makes it possible to manually stop a bad staging commit from
> >> reaching master even if it would compile: once you reset staging, no
> >> already running Patchy process will override that decision with a
> >> version of staging that has become stale.
> >
> > If there are multiple patchy processes, you'd hope they all come to
> > the same conclusion.
>
> Patchy processes don't reset staging.  Humans do.  But our various
> patchies run on different platforms with different version libraries.
> That actually has turned out helpful in discovering portability problems
> at times.

How many patchies are there, and on what platforms do they run?

> > I think the code you are talking about is
> >
> >   
> > https://github.com/gperciva/lilypond-extra/blob/master/patches/compile_lilypond_test/__init__.py#L467
> >
> > which doesn't look very advanced to me.
>
> I haven't said anything about it being "very advanced" or impossible to
> implement.  I have said that your proposed example is an
> oversimplification of what we are doing.

Yes, it's an oversimplification that fits in an email so we can
discuss next steps. My question is: are there fundamental features
that you think are missing?

> > 1) we get a reproducible test process, because everyone can use the
> > same base images.
>
> Which makes it less likely that we discover portability problems.  I am
> not sure what problem you are trying to address here.

It means that if CI sees an error (because it does testing on multiple
platforms), it is trivial for me to reproduce that error, and fix it
locally. By contrast, today if there is an error (see the Pango
problem), we have to email back and forth to figure out what is going
on.

> > 2) we can test against different configurations (Pango 1.44 vs. 1.36,
> > GUILE 1.8 vs 2.2) simultaneously, which catches problems like the
> > recent Pango one earlier.
>
> That's definitely an advantage against our more haphazard setup now.  It
> does come at the cost of _everyone_ (or the CI system) having to test
> _all_ pertinent configurations rather than just a personal sampling if
> it is supposed to increase the covered base.

Nobody _has_ to test all configurations. But one _can_.

-- 
Han-Wen Nienhuys - address@hidden - http://www.xs4all.nl/~hanwen