Re: testing out Docker CI scripts?

[Han-Wen Nienhuys]

On Sun, Feb 23, 2020 at 12:16 AM David Kastrup <address@hidden> wrote:
> >> Which makes it less likely that we discover portability problems.  I
> >> am not sure what problem you are trying to address here.
> >
> > It means that if CI sees an error (because it does testing on multiple
> > platforms), it is trivial for me to reproduce that error, and fix it
> > locally.
>
> The thing with "multiple platforms" is that our testing does not
> actually cover multiple platforms.  The serious testing happens after
> installers are released.  The most release-critical testing is GUB going
> through.  The binaries and installers coming out of GUB never get to see
> a single regtest except possibly manually.

That is true, but I'm trying to argue that the state of Linux testing
is not where it should be, and I am proposing a practical mechanism to
improve it.

It's not perfect, but it's better than what we have.

> I see absolutely no chance that we can change that significantly without
> leaving both the free and the affordable tiers of CI services.
>
> > By contrast, today if there is an error (see the Pango problem), we
> > have to email back and forth to figure out what is going on.
>
> Yes.  But if every developer tests on the same platform, we will have to
> email back and forth with users to figure out what is going on when the
> stuff does not blow up on our unified platform code.  We have had that
> situation with floating point on Windows (or rather 32bit platforms
> generally) just now.  Windows-only problems are really tricky things.
> So I am skeptical that a unification of test platforms among developers
> will make it easier rather than harder to track down problems among us.

So in summary, there is James and you running slightly different
flavors of Linux, doing testing on pending patches (james) and staging
=> master integrations (you).

This means that errors specific to your system (eg. Pango 1.42) are
caught late, when the staging => master push happens.

When we have a docker-based testing, everyone can easily these flavors
of linux, and they can be tested much earlier, for example when the
contributor prepares a patch. This is also especially important once
GUILE 2 moves to a supported state, because we'll have 2 pretty
different platforms.

Moreover, when we discover some other older or newer flavor of Linux
that we also want to support, we can add a Docker image for that too,
and widen our coverage of platforms.

This doesn't solve the problem of getting reliable windows builds, or
usable windows bug reports. The issue of windows support is unrelated
to how well we test on Linux, so I propose we don't discuss it here or
now.

-- 
Han-Wen Nienhuys - address@hidden - http://www.xs4all.nl/~hanwen