Re: CI container image interference between staging and staging-7.2

[Philippe Mathieu-Daudé]

On 13/9/23 01:07, Stefan Hajnoczi wrote:
> Hi,
> TL;DR Michael: Please check that the staging-7.2 branch has Dan's
> commit e28112d00703abd136e2411d23931f4f891c9244 ("gitlab: stable
> staging branches publish containers in a separate tag").
>
> I couldn't explain a check-cfi-x86_64 failure
> (https://gitlab.com/qemu-project/qemu/-/jobs/5072006964), so I reran
> build-cfi-x86_64 to see if it has an effect on its dependencies.
>
> To my surprise the rerun of build-cfi-x86_64 failed:
> https://gitlab.com/qemu-project/qemu/-/jobs/5072087783
>
> The first run was successful:
> https://gitlab.com/qemu-project/qemu/-/jobs/5071532799
>
> Diffing the output shows that the software versions are different. The
> successful run has Python 3.11.5 and meson 1.0.1 while the failed run
> has Python 3.10.8 and meson 0.63.3.
>
> I think staging and staging-7.2 pipelines are interfering with each
> other. My understanding is that build-cfi-x86_64 uses
> registry.gitlab.com/qemu-project/qemu/qemu/fedora:latest and that
> should be built from fedora:38. Python 3.10.8 is what Fedora 35 uses.
> The staging-7.2 branch's fedora.docker file uses fedora:35.

Isn't it clarified by this commit?

commit d4c7a56539d0bdb6fccf60af94d528613cbc7c4b
Author: Daniel P. Berrangé <berrange@redhat.com>
Date:   Thu Jun 8 17:40:14 2023 +0100

    gitlab: centralize the container tag name

    We use a fixed container tag of 'latest' so that contributors' forks
    don't end up with an ever growing number of containers as they work
    on throwaway feature branches.

    This fixed tag causes problems running CI upstream in stable staging
    branches, however, because the stable staging branch will publish
    old container content that clashes with that needed by primary
    staging branch. This makes it impossible to reliably run CI
    pipelines in parallel in upstream for different staging branches.

    This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to
    change which tag container publishing uses. Initially it can be set
    by contributors as a git push option if they want to override the
    default use of 'latest' eg

      git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish

    this is useful if contributors need to run pipelines for different
    branches concurrently in their forks.

    Reviewed-by: Michael Tokarev <mjt@tls.msk.ru>
    Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
    Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
    Message-Id: <20230608164018.2520330-2-berrange@redhat.com>
    Signed-off-by: Thomas Huth <thuth@redhat.com>

Michael, did you forgot to set QEMU_CONTAINER_TAG? Maybe we can add
a tiny scripts/make-stable-release to not forget?