Re: [RFC] Automatic 'make check' in CI

[Jonas Hahnfeld]

Am Donnerstag, den 19.11.2020, 07:28 -0500 schrieb Dan Eble:
> On Nov 19, 2020, at 02:30, Jonas Hahnfeld <hahnjo@hahnjo.de> wrote:
> > 
> > Am Mittwoch, den 18.11.2020, 21:49 -0500 schrieb Dan Eble:
> > > On Nov 18, 2020, at 15:25, Jonas Hahnfeld <hahnjo@hahnjo.de> wrote:
> > > > 
> > > > Until then, we need to rebase to adapt the MR commits to the
> > > > available test-baseline.
> > > 
> > > Do you mean that we would need to rebase to master before any push that 
> > > updates an MR?
> > 
> > At the very least if there were changes to regression tests added to
> > master in the mean time and you don't want them to appear as in your
> > run of "make check". So always doing this should be a safe default
> > (even if this renders the "Compare with previous version" kind of
> > useless; some compromise we have to make here…)
> 
> Hold on.  I understand that "Compare with previous version" becomes
> harder to read after a rebase, but is that the only reason you've
> been thinking of testing against the merge base?

No, the primary reason is reproducibility: If always downloading from
master, you cannot re-run a job on the same input. The merge base stays
constant for a given commit, so you can run the job a second time (if
there was an error on the runner for example) or even restart the
entire pipeline and still get the same test-baseline (or an error if
it's not available anymore).

> I think I would prefer testing against the current version of master.
> That will expose a patch that lags behind as soon as it goes into
> Patch::new rather than delaying until the submitter chooses to rebase
> it, which might be immediately before merging it when nobody is
> paying attention.

Yes, I agree on that. What we could do is enabling Pipelines for Merged
Results, which is a Premium feature (but we have it as OSS project):
https://docs.gitlab.com/ee/ci/merge_request_pipelines/pipelines_for_merged_results/
This simulates a (temporary) merge to master at the time the pipeline
starts. I don't like it very much because we have a linear history in
master, so the final pipeline will run on a temporary merge commit that
will never actually be part of the repository. GitLab is working on
enabling the rebase strategy for this (so it would temporarily rebase
your branch on master and run the pipeline), but it's not there yet...
(Btw this would also give us a working merge train when multiple MRs
count down on the same day!)

Jonas

signature.asc
Description: This is a digitally signed message part