In addition, with the expected change in GitLab to set a finite CI
minute allowance on all users we need to be much more conservative.
This patch thus sets up rules so that no CI jobs will run by default
on push, except
- Pushes to branch name prefix 'staging' on 'qemu-project' repo
run full set of jobs
- Pushes to 'master' branch, only run jobs needed to publish
the website.
In any pushes to users forks, CI is now strictly opt-in. The pipeline
will always be created by every singe job will be marked manual. You
can then use the web UI to start individual jobs.
For more convenience there are three levels of increasing comprehensive
CI job sets that can be chosen between
- Gating - run exactly the same as pushes to 'staging' branch
(except for jobs needing custom runners)
Push to 'ci-gating-xxx' branch, or set env QEMU_CI_GATING=1
- Full - same as 'Gating', except that acceptance tests don't
get run. This is equivalent to historical CI behaviour
for pushes to user forks.
Push to 'ci-full-xxx' branch, or set env QEMU_CI_FULL=1
- Minimal - a significantly cut down set of jobs to get a
decent sanity check of builds without burning
massive amounts of CI time.
Push to 'ci-min-xxx' branch, or set env QEMU_CI=1
The minimal job set covers:
* Fedora, CentOS, Ubuntu & Debian system emulator builds
to cover all arch targets.
* Linux user static build
* Cross build i386 (to identify any 32-bit build issues)
* Cross build s390x (to identify any big endian build issues)
* Containers needed for the above
* Cirrus CI FreeBSD 12 and macOS 11 (to identify non-Linux issues)
* Simple checks - python unittests, DCO check, checkpatch.pl, etc
This gives about 30 jobs instead of 120 from the "Full" group. It
is likely reasonable to cut the minimal set down even more, as IMHO
there are too many system emulator jobs there.
This is all controlled by defining a set of 'rules' entries that
apply to (almost) all jobs in the CI pipeline. The exceptions are
those using custom runners, and the firmware build jobs.
Using inherited templates together with rules, doesn't work in
the way you might expect. Any rules defined in the child completely
replace those from the parent, rather than augmenting them. Thus we
need to avoid having rules defined in any jobs at all, by inheriting
from a parent job template that contains the rules. The characteristics
of individual jobs are then defined by variables set against the job.
This is described in the updated content for the docs/devel/ci-jobs.rst
file.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
.gitlab-ci.d/buildtest-template.yml | 17 +--
.gitlab-ci.d/buildtest.yml | 43 ++++----
.gitlab-ci.d/cirrus.yml | 6 +-
.gitlab-ci.d/container-core.yml | 4 +
.gitlab-ci.d/container-cross.yml | 9 +-
.gitlab-ci.d/container-template.yml | 1 +
.gitlab-ci.d/containers.yml | 4 +
.gitlab-ci.d/crossbuild-template.yml | 3 +
.gitlab-ci.d/crossbuilds.yml | 3 +
.gitlab-ci.d/qemu-project.yml | 1 +
.gitlab-ci.d/rules.yml | 116 ++++++++++++++++++++
.gitlab-ci.d/static_checks.yml | 18 ++--
.gitlab-ci.yml | 4 -
docs/devel/ci-jobs.rst | 155 ++++++++++++++++++++++++---
14 files changed, 317 insertions(+), 67 deletions(-)
create mode 100644 .gitlab-ci.d/rules.yml
diff --git a/.gitlab-ci.d/buildtest-template.yml
b/.gitlab-ci.d/buildtest-template.yml
index fcbcc4e627..8578b3ed11 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -1,4 +1,6 @@
+
.native_build_job_template:
+ extends: .job_rules
stage: build
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
before_script:
@@ -27,6 +29,7 @@
fi
.native_test_job_template:
+ extends: .job_rules
stage: test
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
script:
@@ -67,15 +70,5 @@
after_script:
- cd build
- du -chs ${CI_PROJECT_DIR}/avocado-cache
- rules:
- # Only run these jobs if running on the mainstream namespace,
- # or if the user set the QEMU_CI_AVOCADO_TESTING variable (either
- # in its namespace setting or via git-push option, see documentation
- # in /.gitlab-ci.yml of this repository).
- - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
- when: on_success
- - if: '$QEMU_CI_AVOCADO_TESTING'
- when: on_success
- # Otherwise, set to manual (the jobs are created but not run).
- - when: manual
- allow_failure: true
+ variables:
+ QEMU_JOB_GATING: 1
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 903ee65f32..c8e03373c5 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -40,6 +40,7 @@ build-system-ubuntu:
needs:
job: amd64-ubuntu2004-container
variables:
+ QEMU_JOB_MINIMAL: 1
IMAGE: ubuntu2004
CONFIGURE_ARGS: --enable-docs --enable-fdt=system --enable-slirp=system
TARGETS: aarch64-softmmu alpha-softmmu cris-softmmu hppa-softmmu
@@ -56,6 +57,7 @@ check-system-ubuntu:
- job: build-system-ubuntu
artifacts: true
variables:
+ QEMU_JOB_MINIMAL: 1
IMAGE: ubuntu2004
MAKE_CHECK_ARGS: check
@@ -73,6 +75,7 @@ build-system-debian:
needs:
job: amd64-debian-container
variables:
+ QEMU_JOB_MINIMAL: 1
IMAGE: debian-amd64
CONFIGURE_ARGS: --enable-fdt=system
TARGETS: arm-softmmu avr-softmmu i386-softmmu mipsel-softmmu
@@ -89,6 +92,7 @@ check-system-debian:
- job: build-system-debian
artifacts: true
variables:
+ QEMU_JOB_MINIMAL: 1
IMAGE: debian-amd64
MAKE_CHECK_ARGS: check
@@ -106,6 +110,7 @@ build-system-fedora:
needs:
job: amd64-fedora-container
variables:
+ QEMU_JOB_MINIMAL: 1
IMAGE: fedora
CONFIGURE_ARGS: --disable-gcrypt --enable-nettle --enable-docs
--enable-fdt=system --enable-slirp=system
--enable-capstone=system
@@ -123,6 +128,7 @@ check-system-fedora:
- job: build-system-fedora
artifacts: true
variables:
+ QEMU_JOB_MINIMAL: 1
IMAGE: fedora
MAKE_CHECK_ARGS: check
@@ -140,6 +146,7 @@ build-system-centos:
needs:
job: amd64-centos8-container
variables:
+ QEMU_JOB_MINIMAL: 1
IMAGE: centos8
CONFIGURE_ARGS: --disable-nettle --enable-gcrypt --enable-fdt=system
--enable-modules --enable-trace-backends=dtrace
@@ -157,6 +164,7 @@ check-system-centos:
- job: build-system-centos
artifacts: true
variables:
+ QEMU_JOB_MINIMAL: 1
IMAGE: centos8
MAKE_CHECK_ARGS: check
@@ -325,6 +333,7 @@ build-user-static:
needs:
job: amd64-debian-user-cross-container
variables:
+ QEMU_JOB_MINIMAL: 1
IMAGE: debian-all-test-cross
CONFIGURE_ARGS: --disable-tools --disable-system --static
MAKE_CHECK_ARGS: check-tcg
@@ -411,17 +420,15 @@ build-cfi-aarch64:
--enable-safe-stack --enable-slirp=git
TARGETS: aarch64-softmmu
MAKE_CHECK_ARGS: check-build
+ # FIXME: This job is often failing, likely due to out-of-memory problems in
+ # the constrained containers of the shared runners. Thus this is marked as
+ # manual until the situation has been solved.
+ QEMU_JOB_MANUAL: 1
timeout: 70m
artifacts:
expire_in: 2 days
paths:
- build
- rules:
- # FIXME: This job is often failing, likely due to out-of-memory problems in
- # the constrained containers of the shared runners. Thus this is marked as
- # manual until the situation has been solved.
- - when: manual
- allow_failure: true
check-cfi-aarch64:
extends: .native_test_job_template
@@ -453,17 +460,15 @@ build-cfi-ppc64-s390x:
--enable-safe-stack --enable-slirp=git
TARGETS: ppc64-softmmu s390x-softmmu
MAKE_CHECK_ARGS: check-build
+ # FIXME: This job is often failing, likely due to out-of-memory problems in
+ # the constrained containers of the shared runners. Thus this is marked as
+ # manual until the situation has been solved.
+ QEMU_JOB_MANUAL: 1
timeout: 70m
artifacts:
expire_in: 2 days
paths:
- build
- rules:
- # FIXME: This job is often failing, likely due to out-of-memory problems in
- # the constrained containers of the shared runners. Thus this is marked as
- # manual until the situation has been solved.
- - when: manual
- allow_failure: true
check-cfi-ppc64-s390x:
extends: .native_test_job_template
@@ -657,6 +662,7 @@ build-without-default-features:
MAKE_CHECK_ARGS: check-unit
build-libvhost-user:
+ extends: .job_rules
stage: build
image: $CI_REGISTRY_IMAGE/qemu/fedora:latest
needs:
@@ -676,6 +682,8 @@ build-tools-and-docs-debian:
needs:
job: amd64-debian-container
variables:
+ QEMU_JOB_MINIMAL: 1
+ QEMU_JOB_PUBLISH: 1
IMAGE: debian-amd64
MAKE_CHECK_ARGS: check-unit check-softfloat ctags TAGS cscope
CONFIGURE_ARGS: --disable-system --disable-user --enable-docs
--enable-tools
@@ -698,10 +706,14 @@ build-tools-and-docs-debian:
# that users can see the results of their commits, regardless
# of what topic branch they're currently using
pages:
+ extends: .native_build_job_template
image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest
stage: test
needs:
- job: build-tools-and-docs-debian
+ variables:
+ QEMU_JOB_MINIMAL: 1
+ QEMU_JOB_PUBLISH: 1
script:
- mkdir -p public
# HTML-ised source tree
@@ -715,10 +727,3 @@ pages:
artifacts:
paths:
- public
- rules:
- - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH ==
$CI_DEFAULT_BRANCH'
- when: on_success
- - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
- when: never
- - if: '$CI_PROJECT_NAMESPACE != "qemu-project"'
- when: on_success
diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml
index 675db69622..4334736796 100644
--- a/.gitlab-ci.d/cirrus.yml
+++ b/.gitlab-ci.d/cirrus.yml
@@ -11,6 +11,7 @@
# special care, because we can't just override it at the GitLab CI job
# definition level or we risk breaking it completely.
.cirrus_build_job:
+ extends: .job_rules
stage: build
image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master
needs: []
@@ -39,8 +40,6 @@
<.gitlab-ci.d/cirrus/build.yml >.gitlab-ci.d/cirrus/$NAME.yml
- cat .gitlab-ci.d/cirrus/$NAME.yml
- cirrus-run -v --show-build-log always .gitlab-ci.d/cirrus/$NAME.yml
- rules:
- - if: "$CIRRUS_GITHUB_REPO && $CIRRUS_API_TOKEN"
x64-freebsd-12-build:
extends: .cirrus_build_job
@@ -61,6 +60,8 @@ x64-freebsd-12-build:
x64-freebsd-13-build:
extends: .cirrus_build_job
variables:
+ QEMU_JOB_SOFT_FAIL: 1
+ QEMU_JOB_MINIMAL: 1
NAME: freebsd-13
CIRRUS_VM_INSTANCE_TYPE: freebsd_instance
CIRRUS_VM_IMAGE_SELECTOR: image_family
@@ -74,6 +75,7 @@ x64-freebsd-13-build:
x64-macos-11-base-build:
extends: .cirrus_build_job
variables:
+ QEMU_JOB_MINIMAL: 1
NAME: macos-11
CIRRUS_VM_INSTANCE_TYPE: osx_instance
CIRRUS_VM_IMAGE_SELECTOR: image
diff --git a/.gitlab-ci.d/container-core.yml b/.gitlab-ci.d/container-core.yml
index e8dd1f476a..8983c32b3b 100644
--- a/.gitlab-ci.d/container-core.yml
+++ b/.gitlab-ci.d/container-core.yml
@@ -4,14 +4,18 @@ include:
amd64-centos8-container:
extends: .container_job_template
variables:
+ QEMU_JOB_MINIMAL: 1
NAME: centos8
amd64-fedora-container:
extends: .container_job_template
variables:
+ QEMU_JOB_MINIMAL: 1
NAME: fedora
amd64-debian10-container:
extends: .container_job_template
variables:
+ QEMU_JOB_MINIMAL: 1
+ QEMU_JOB_PUBLISH: 1
NAME: debian10
diff --git a/.gitlab-ci.d/container-cross.yml b/.gitlab-ci.d/container-cross.yml
index 0fcebe363a..ef316668eb 100644
--- a/.gitlab-ci.d/container-cross.yml
+++ b/.gitlab-ci.d/container-cross.yml
@@ -50,13 +50,11 @@ armhf-debian-cross-container:
# We never want to build hexagon in the CI system and by default we
# always want to refer to the master registry where it lives.
hexagon-cross-container:
+ extends: .job_rules
image: docker:stable
stage: containers
- rules:
- - if: '$CI_PROJECT_NAMESPACE == "qemu-project"'
- when: never
- - when: always
variables:
+ QEMU_JOB_MANUAL: 1
NAME: debian-hexagon-cross
GIT_DEPTH: 1
services:
@@ -143,6 +141,7 @@ s390x-debian-cross-container:
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
+ QEMU_JOB_MINIMAL: 1
NAME: debian-s390x-cross
sh4-debian-cross-container:
@@ -179,11 +178,13 @@ cris-fedora-cross-container:
i386-fedora-cross-container:
extends: .container_job_template
variables:
+ QEMU_JOB_MINIMAL: 1
NAME: fedora-i386-cross
win32-fedora-cross-container:
extends: .container_job_template
variables:
+ QEMU_JOB_MINIMAL: 1
NAME: fedora-win32-cross
win64-fedora-cross-container:
diff --git a/.gitlab-ci.d/container-template.yml
b/.gitlab-ci.d/container-template.yml
index 1baecd9460..e2954e4ed3 100644
--- a/.gitlab-ci.d/container-template.yml
+++ b/.gitlab-ci.d/container-template.yml
@@ -1,4 +1,5 @@
.container_job_template:
+ extends: .job_rules
image: docker:stable
stage: containers
services:
diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
index cd06d3f5f4..6948923892 100644
--- a/.gitlab-ci.d/containers.yml
+++ b/.gitlab-ci.d/containers.yml
@@ -17,6 +17,8 @@ amd64-debian-container:
stage: containers-layer2
needs: ['amd64-debian10-container']
variables:
+ QEMU_JOB_MINIMAL: 1
+ QEMU_JOB_PUBLISH: 1
NAME: debian-amd64
amd64-ubuntu1804-container:
@@ -27,6 +29,7 @@ amd64-ubuntu1804-container:
amd64-ubuntu2004-container:
extends: .container_job_template
variables:
+ QEMU_JOB_MINIMAL: 1
NAME: ubuntu2004
amd64-ubuntu-container:
@@ -42,4 +45,5 @@ amd64-opensuse-leap-container:
python-container:
extends: .container_job_template
variables:
+ QEMU_JOB_MINIMAL: 1
NAME: python
diff --git a/.gitlab-ci.d/crossbuild-template.yml
b/.gitlab-ci.d/crossbuild-template.yml
index 7d3ad00a1e..997ec02f79 100644
--- a/.gitlab-ci.d/crossbuild-template.yml
+++ b/.gitlab-ci.d/crossbuild-template.yml
@@ -1,4 +1,5 @@
.cross_system_build_job:
+ extends: .job_rules
stage: build
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
timeout: 80m
@@ -23,6 +24,7 @@
# KVM), and set extra options (such disabling other accelerators) via the
# $EXTRA_CONFIGURE_OPTS variable.
.cross_accel_build_job:
+ extends: .job_rules
stage: build
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
timeout: 30m
@@ -35,6 +37,7 @@
- make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
.cross_user_build_job:
+ extends: .job_rules
stage: build
image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest
script:
diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index 4ff3aa3cfc..9fcd2ac567 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -48,6 +48,7 @@ cross-i386-system:
needs:
job: i386-fedora-cross-container
variables:
+ QEMU_JOB_MINIMAL: 1
IMAGE: fedora-i386-cross
MAKE_CHECK_ARGS: check-qtest
@@ -129,6 +130,7 @@ cross-s390x-system:
needs:
job: s390x-debian-cross-container
variables:
+ QEMU_JOB_MINIMAL: 1
IMAGE: debian-s390x-cross
cross-s390x-user:
@@ -159,6 +161,7 @@ cross-win32-system:
needs:
job: win32-fedora-cross-container
variables:
+ QEMU_JOB_MINIMAL: 1
IMAGE: fedora-win32-cross
artifacts:
paths:
diff --git a/.gitlab-ci.d/qemu-project.yml b/.gitlab-ci.d/qemu-project.yml
index b3d79bc429..7007193fc0 100644
--- a/.gitlab-ci.d/qemu-project.yml
+++ b/.gitlab-ci.d/qemu-project.yml
@@ -2,6 +2,7 @@
# https://gitlab.com/qemu-project/qemu/-/pipelines
include:
+ - local: '/.gitlab-ci.d/rules.yml'
- local: '/.gitlab-ci.d/stages.yml'
- local: '/.gitlab-ci.d/edk2.yml'
- local: '/.gitlab-ci.d/opensbi.yml'
diff --git a/.gitlab-ci.d/rules.yml b/.gitlab-ci.d/rules.yml
new file mode 100644
index 0000000000..3399722ca9
--- /dev/null
+++ b/.gitlab-ci.d/rules.yml
@@ -0,0 +1,116 @@
+
+# This defines rules used to control individual job execution
+# See docs/devel/ci-jobs.rst for an explanation of the various
+# variables and branch naming conventions applied here.
+#
+.job_rules:
+ rules:
+ # ======================================================================
+ # Rules that apply regardless of whether the primary qemu repo or a fork
+ # ======================================================================
+
+ # Skip any cirrus job if either repo or api token are missing
+ # as we won't be able to talk to cirrus
+ - if: '$CIRRUS_VM_INSTANCE_TYPE && ($CIRRUS_GITHUB_REPO == null ||
$CIRRUS_API_TOKEN == null)'
+ when: never
+
+ # Any jobs marked as manual, never get automatically run in any scenario
+ # and don't contribute to pipeline status
+ - if: '$QEMU_JOB_MANUAL'
+ when: manual
+ allow_failure: true
+
+ # For the main repo, tags represent official releases.
+ # The branch associated with the release will have passed
+ # a CI pipeline already
+ #
+ # For user forks, tags are tyically added by tools like
+ # git-publish at the same time as pushing the branch prior
+ # to sending out for review
+ #
+ # In neither case do we wish to run CI pipelines for tags
+ - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_TAG'
+ when: never
+
+
+ # ==================================================
+ # Rules to control what happens in primary qemu repo
+ # ==================================================
+
+ # The 'pages' job must never run on a non-default branch as we don't
+ # want to publish the website from pre-merged content
+ - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" &&
$CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_JOB_NAME == "pages"'
+ when: never
+
+ # Run all jobs for main repo staging branch
+ - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" &&
$CI_COMMIT_BRANCH == "staging"'
+ when: on_success
+
+ # Run jobs needed to support website publishing on the main branch
+ - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" &&
$CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $QEMU_JOB_PUBLISH'
+ when: on_success
+
+ # Prevent jobs from running on pushes to any other branch / tag
+ - if: '$CI_PIPELINE_SOURCE == "push" && $CI_PROJECT_NAMESPACE == "qemu-project" &&
$CI_COMMIT_BRANCH != "staging"'
+ when: never
+
+
+ # Remainining pipeline sources follow the same rules as for fork repos
+
+
+
+ # ====================================
+ # Rules for running jobs in user forks
+ # ====================================
+
+ # Part 1: gating jobs
+ # -------------------
+
+ # If on a branch with name prefix 'ci-acceptance-', then run
+ # everything, just as a gating job on 'staging' branch would
+ - if: '$CI_COMMIT_BRANCH =~ /^ci-gating-/'
+ when: on_success
+
+ # If user set QEMU_CI_GATING=1, then run everything just as
+ # a gating job on 'staging' branch would
+ - if: '$QEMU_CI_GATING'
+ when: on_success
+
+ # Otherwise never run jobs marked as gating, but allow manual trigger
+ # without affecting pipeline status
+ - if: '$QEMU_JOB_GATING'
+ when: manual
+ allow_failure: true
+
+
+ # =============================================
+ # Part 2: opt-in for all CI, except gating jobs
+ # =============================================
+
+ # If pushing to a branch with name prefix 'ci-all', then run all jobs
+ - if: '$CI_COMMIT_BRANCH =~ /^ci-all/'
+ when: on_success
+
+ # If user QEMU_CI_ALL=1, then run all jobs
+ - if: '$QEMU_CI_ALL'
+ when: on_success
+
+
+ # ===============================
+ # Part 3: the minimal set of jobs
+ # ===============================
+
+ # If pushing to a branch with name prefix 'ci-min', then run
+ # only a minimal set of jobs
+ - if: '$CI_COMMIT_BRANCH =~ /^ci-min/ && $QEMU_JOB_MINIMAL'
+ when: on_success
+
+ # If user set QEMU_CI=1, then run only a minimal set of jobs
+ - if: '$QEMU_CI && $QEMU_JOB_MINIMAL'
+ when: on_success
+
+ # Any other scenario, let the jobs be manually triggered only.
+ # Set to be non-fatal, so that pending manual jobs don't
+ # affect overall pipeline status
+ - when: manual
+ allow_failure: true
diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml
index 96dbd9e310..4ba3e038dd 100644
--- a/.gitlab-ci.d/static_checks.yml
+++ b/.gitlab-ci.d/static_checks.yml
@@ -1,4 +1,5 @@
check-patch:
+ extends: .job_rules
stage: build
image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
needs:
@@ -6,42 +7,41 @@ check-patch:
script:
- .gitlab-ci.d/check-patch.py
variables:
+ QEMU_JOB_MINIMAL: 1
GIT_DEPTH: 1000
- rules:
- - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH ==
$CI_DEFAULT_BRANCH'
- when: never
- - when: on_success
- allow_failure: true
+ allow_failure: true
check-dco:
+ extends: .job_rules
stage: build
image: $CI_REGISTRY_IMAGE/qemu/centos8:latest
needs:
job: amd64-centos8-container
script: .gitlab-ci.d/check-dco.py
variables:
+ QEMU_JOB_MINIMAL: 1
GIT_DEPTH: 1000
- rules:
- - if: '$CI_PROJECT_NAMESPACE == "qemu-project" && $CI_COMMIT_BRANCH ==
$CI_DEFAULT_BRANCH'
- when: never
- - when: on_success
check-python-pipenv:
+ extends: .job_rules
stage: test
image: $CI_REGISTRY_IMAGE/qemu/python:latest
script:
- make -C python check-pipenv
variables:
+ QEMU_JOB_MINIMAL: 1
GIT_DEPTH: 1
needs:
job: python-container
check-python-tox:
+ extends: .job_rules
stage: test
image: $CI_REGISTRY_IMAGE/qemu/python:latest
script:
- make -C python check-tox
variables:
+ QEMU_JOB_MINIMAL: 1
GIT_DEPTH: 1
QEMU_TOX_EXTRA_ARGS: --skip-missing-interpreters=false
needs:
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 9762dda2ee..d5bd31f294 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -15,10 +15,6 @@
#
# QEMU CI jobs are based on templates. Some templates provide
# user-configurable options, modifiable via configuration variables.
-#
-# See https://qemu-project.gitlab.io/qemu/devel/ci.html#custom-ci-cd-variables
-# for more information.
-#
include:
- local: '/.gitlab-ci.d/qemu-project.yml'
diff --git a/docs/devel/ci-jobs.rst b/docs/devel/ci-jobs.rst
index 9cd9819786..6d3f13d495 100644
--- a/docs/devel/ci-jobs.rst
+++ b/docs/devel/ci-jobs.rst
@@ -1,22 +1,49 @@
-Custom CI/CD variables
-======================
+CI Job Rules
+============
-QEMU CI pipelines can be tuned by setting some CI environment variables.
+The following documents how CI rules control execution of jobs in a pipeline
-Set variable globally in the user's CI namespace
-------------------------------------------------
+Job variable naming
+-------------------
-Variables can be set globally in the user's CI namespace setting.
+The rules for controlling jobs in a pipeline will be driven by a number of
+variables:
+
+ - ``CI_XXX``, ``GITLAB_XXX`` - variables defined by GitLab:
+
+ https://docs.gitlab.com/ee/ci/variables/predefined_variables.html
+
+ - ``QEMU_JOB_XXX`` - variables defined in the YAML files that express
+ characteristics of the job used to control default behaviour
+
+ - ``QEMU_CI_XXX`` - variables defined by the user that are used to fine
+ tune which jobs are run dynamically
+
+and in some cases based on the branch name prefixes.
+
+Job fine tuning strategies
+--------------------------
+
+Using a combination of the ``QEMU_CI_XXX`` variables and ``ci-``
+branch name prefix, users can fine tune what jobs are run.
+
+Set variable globally in the gitlab repository CI config
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Variables can be set globally in the user's gitlab repository CI config.
+These variables will apply to all pipelines associated with the repository
+thereafter. This is useful for fine tuning the jobs indefinitely.
For further information about how to set these variables, please refer to::
https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
-Set variable manually when pushing a branch or tag to the user's repository
----------------------------------------------------------------------------
+Set variable manually when pushing to git
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Variables can be set manually when pushing a branch or tag, using
-git-push command line arguments.
+git-push command line arguments. This is useful for fine tuning the
+jobs on an adhoc basis.
Example setting the QEMU_CI_EXAMPLE_VAR variable:
@@ -28,13 +55,107 @@ For further information about how to set these variables, please refer to::
https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
-Here is a list of the most used variables:
+Pushing to specially named branches
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Branch names starting with a 'ci-' prefix can be used as an alternative
+to setting variables. Details of supported branch names are detailed
+later in this document.
+
+Provide a custom gitlab CI configuration
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The above strategies all provide a way to fine tune the jobs defined by the
+standard QEMU gitlab CI configuration file. If this is not sufficient it is
+possible to completely replace the CI configuration file with a custom
+version. This allows the contributor to achieve essentially anything they
+desire, within the scope of what GitLab supports.
+
+Replacing the ``.gitlab-ci.yml`` file is done in the repository settings:
+
+
https://docs.gitlab.com/ee/ci/pipelines/settings.html#specify-a-custom-cicd-configuration-file
+
+Note that it is possible to reference an arbitrary URL. This could point
+to a replacement .gitlab-ci.yml on a specific branch in the same repo,
+or can point to any external site.
+
+
+Job grouping sets
+-----------------
+
+There are many different jobs defined in the GitLab CI pipeline used by QEMU.
+It is not practical to run all jobs in every scenario, so there are a set of
+rules defined that control which jobs are executed on each pipeline run. At
+a high level the jobs can be grouped into a number of mutually exclusive
+sets.
+
+ - Manual jobs
+
+ This is a set of jobs that will never be run automatically in any scenario.
+ The reason a job will be marked manual is if it is known to exhibit some
+ non-deterministic failures, or liable to trigger timeouts. Ideally jobs are
+ only present in this set for a short period of time while the problems are
+ investigated and resolved. Users can manually trigger these jobs from the
+ the pipelines web UI if desired, but they will never contribute to the
+ overall pipeline status.
+
+ Identified by the variable ``QEMU_JOB_MANUAL: 1``
+
+
+ - Minimal jobs
+
+ This is a minimal set of jobs that give a reasonable build and test sanity
+ check, which will be run by default in all possible scenarios. This set of
+ jobs is intended to be controlled fairly strictly to avoid wasting CI
+ minutes of maintainers/contributors. The intent is to get 80-90% coverage
+ of build and unit tests across the most important host platforms and
+ architectures.
+
+ Identified by the variable ``QEMU_JOB_MINIMAL: 1``
+
+ Run by setting the variable ``QEMU_CI: 1`` or pushing to a branch
+ named ``ci-min-XXX``.
+
+
+ - Gating jobs
+
+ This is a set of jobs will will always run as a gating test prior to code
+ merging into the default branch of the primary QEMU git repository. In user
+ forks the jobs will not run by default, but the user can opt-in to trigger
+ their execution. These jobs may run a particularly thorough set of scenarios
+ that maintainers are not normally going to exercise before sending series.
+
+ Identified by the variable ``QEMU_JOB_GATING: 1``
+
+ Run by setting the variable ``QEMU_CI_GATING: 1`` or pushing to a branch
+ named ``ci-gating-XXX``
+
+
+ - Optional jobs
+
+ This is all remaining jobs that don't fall into one of the above buckets.
+ These jobs will always be run as a gating test prior to code merging into
+ the default branch of the primary QEMU git repository. In user forks the
+ jobs will not run by default, but the user can opt-in to trigger their
+ execution. These jobs will aim to expand the host/platform build coverage
+ gaps left by the default jobs.
+
+ Identified by not having any of the above variables set.
+
+ Run by setting the variable ``QEMU_CI_ALL: 1`` or pushing to a branch
+ named ``ci-all-XXXX``
+
+Since the sets are mutually exclusive, if more than one of the variables above
+is set, then only one of them will be honoured. The precedence matches the
+ordering of the above docs.
+
+In addition to the above, there are some special extra sets that can augment
+any of the above sets
+
+ - Publishing jobs
-QEMU_CI_AVOCADO_TESTING
-~~~~~~~~~~~~~~~~~~~~~~~
-By default, tests using the Avocado framework are not run automatically in
-the pipelines (because multiple artifacts have to be downloaded, and if
-these artifacts are not already cached, downloading them make the jobs
-reach the timeout limit). Set this variable to have the tests using the
-Avocado framework run automatically.
+ These are jobs that are needed to publish the QEMU website. For user fork
+ repos these follow the normal rules set out above. For the primary QEMU
+ repo, however, the jobs will also run on the default branch.
+ Identified by the variable ``QEMU_JOB_PUBLISH: 1``