Re: testing out Docker CI scripts?

[Han-Wen Nienhuys]

On Sun, Feb 23, 2020 at 12:16 AM David Kastrup <address@hidden> wrote:
> > Yes, it's an oversimplification that fits in an email so we can
> > discuss next steps. My question is: are there fundamental features
> > that you think are missing?
>
> In what we are doing now?  Or in what you have proposed?  Since the
> latter apparently is to include oversimplification, I don't see how I
> could answer that without actually seeing the full version.

Maybe you could play around with it for a bit, and see if you think it
does something useful?

Honestly, we all have agreed that the current infrastructure is
crumbling, yet you need you need to see the completely rewritten
infrastructure before you can decide it's worth it? That is a pretty
tall order to ask of potential contributors.

The scripts I am providing provide a concise way of getting test
results for multiple platforms. I can write some more automation for
patches, but it's hard to tell what to work on next.

> The current version has evolved to do a reasonable job within the
> framework of people running it on their personal setup.  A considerable
> change of framework would obviously trigger the question again, and the

trigger what question again?

> per-patch Patchy operated by James only obviously has sunk to a state
> where fundamental features are not as much missing as having become
> inoperative.
>
> >> > 1) we get a reproducible test process, because everyone can use the
> >> > same base images.
> >>
> >> Which makes it less likely that we discover portability problems.  I
> >> am not sure what problem you are trying to address here.
> >
> > It means that if CI sees an error (because it does testing on multiple
> > platforms), it is trivial for me to reproduce that error, and fix it
> > locally.
>
> The thing with "multiple platforms" is that our testing does not
> actually cover multiple platforms.  The serious testing happens after
> installers are released.  The most release-critical testing is GUB going
> through.  The binaries and installers coming out of GUB never get to see
> a single regtest except possibly manually.
>
> I see absolutely no chance that we can change that significantly without
> leaving both the free and the affordable tiers of CI services.
>
> > By contrast, today if there is an error (see the Pango problem), we
> > have to email back and forth to figure out what is going on.
>
> Yes.  But if every developer tests on the same platform, we will have to
> email back and forth with users to figure out what is going on when the
> stuff does not blow up on our unified platform code.  We have had that
> situation with floating point on Windows (or rather 32bit platforms
> generally) just now.  Windows-only problems are really tricky things.
> So I am skeptical that a unification of test platforms among developers
> will make it easier rather than harder to track down problems among us.
>
> >> > 2) we can test against different configurations (Pango 1.44
> >> > vs. 1.36, GUILE 1.8 vs 2.2) simultaneously, which catches problems
> >> > like the recent Pango one earlier.
> >>
> >> That's definitely an advantage against our more haphazard setup now.
> >> It does come at the cost of _everyone_ (or the CI system) having to
> >> test _all_ pertinent configurations rather than just a personal
> >> sampling if it is supposed to increase the covered base.
> >
> > Nobody _has_ to test all configurations. But one _can_.
>
> If one does, the bill will come up eventually.  For better or worse,
> LilyPond is a real pig regarding resource usage for full builds/tests.
> Our strategy so far has been working in a spotty manner, and with
> volunteers giving their computers significant workouts.
>
> That's not how you would do things in a corporate setting.  But we don't
> have a corporate setting.
>
> --
> David Kastrup
> My replies have a tendency to cause friction.  To help mitigating
> damage, feel free to forward problematic posts to me adding a subject
> like "timeout 1d" (for a suggested timeout of 1 day) or "offensive".



-- 
Han-Wen Nienhuys - address@hidden - http://www.xs4all.nl/~hanwen