Re: [PATCH] Remove Ubuntu 18.04 support from the repository

[Daniel P . Berrangé]

On Tue, May 10, 2022 at 09:56:12PM +0200, Thomas Huth wrote:
> According to our "Supported build platforms" policy, we now do not support
> Ubuntu 18.04 anymore. Remove the related files and entries from our CI.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  Seems like nobody touched the 18.04-based tests/vm/ubuntu* files in a
>  very long time, so I assume these are not used anymore and can completely
>  be removed now.

Or it could mean that they are working fine and so haven't needed
changes...

The cross-compiler container envs give us build coverage for
non-x86, but the VMs give developers on x86 the ability to
actually run the unit tests / I/O tests. Personally I've
never used them, but someone might be ?

I'd suggest splitting this patch into several pieces. One
to fixup all the custom runner related pieces to remove
references to 18.04. One to delete the 18.04 container
related pieces. One to delete the 18.04 VM related pieces.

If you're adventurous, add a 22.04 container since that
is released now and in the QEMU platform target scope.

> 
>  .gitlab-ci.d/containers.yml                   |   5 -
>  .../custom-runners/ubuntu-20.04-aarch32.yml   |   2 +-
>  .../custom-runners/ubuntu-20.04-aarch64.yml   |   2 +-
>  scripts/ci/setup/build-environment.yml        |  14 +-
>  tests/docker/dockerfiles/ubuntu1804.docker    | 144 ------------------
>  tests/lcitool/refresh                         |   7 -
>  tests/vm/ubuntu.aarch64                       |  68 ---------
>  tests/vm/ubuntu.i386                          |  40 -----
>  tests/vm/ubuntuvm.py                          |  60 --------
>  9 files changed, 4 insertions(+), 338 deletions(-)
>  delete mode 100644 tests/docker/dockerfiles/ubuntu1804.docker
>  delete mode 100755 tests/vm/ubuntu.aarch64
>  delete mode 100755 tests/vm/ubuntu.i386
>  delete mode 100644 tests/vm/ubuntuvm.py
> 
> diff --git a/.gitlab-ci.d/containers.yml b/.gitlab-ci.d/containers.yml
> index b9b675fdcb..e9df90bbdd 100644
> --- a/.gitlab-ci.d/containers.yml
> +++ b/.gitlab-ci.d/containers.yml
> @@ -19,11 +19,6 @@ amd64-debian-container:
>    variables:
>      NAME: debian-amd64
>  
> -amd64-ubuntu1804-container:
> -  extends: .container_job_template
> -  variables:
> -    NAME: ubuntu1804
> -
>  amd64-ubuntu2004-container:
>    extends: .container_job_template
>    variables:
> diff --git a/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch32.yml 
> b/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch32.yml
> index 9c589bc4cf..47856ac53c 100644
> --- a/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch32.yml
> +++ b/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch32.yml
> @@ -1,6 +1,6 @@
>  # All ubuntu-20.04 jobs should run successfully in an environment
>  # setup by the scripts/ci/setup/qemu/build-environment.yml task
> -# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> +# "Install basic packages to build QEMU on Ubuntu 20.04"
>  
>  ubuntu-20.04-aarch32-all:
>   needs: []
> diff --git a/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch64.yml 
> b/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch64.yml
> index 920e388bd0..951e490db1 100644
> --- a/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch64.yml
> +++ b/.gitlab-ci.d/custom-runners/ubuntu-20.04-aarch64.yml
> @@ -1,6 +1,6 @@
>  # All ubuntu-20.04 jobs should run successfully in an environment
>  # setup by the scripts/ci/setup/qemu/build-environment.yml task
> -# "Install basic packages to build QEMU on Ubuntu 18.04/20.04"
> +# "Install basic packages to build QEMU on Ubuntu 20.04"
>  
>  ubuntu-20.04-aarch64-all-linux-static:
>   needs: []
> diff --git a/scripts/ci/setup/build-environment.yml 
> b/scripts/ci/setup/build-environment.yml
> index 9182e0c253..232525b91d 100644
> --- a/scripts/ci/setup/build-environment.yml
> +++ b/scripts/ci/setup/build-environment.yml
> @@ -33,10 +33,9 @@
>        when:
>          - ansible_facts['distribution'] == 'Ubuntu'
>  
> -    - name: Install basic packages to build QEMU on Ubuntu 18.04/20.04
> +    - name: Install basic packages to build QEMU on Ubuntu 20.04
>        package:
>          name:
> -        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
>            - ccache
>            - gcc
>            - gettext
> @@ -90,7 +89,7 @@
>        when:
>          - ansible_facts['distribution'] == 'Ubuntu'
>  
> -    - name: Install packages to build QEMU on Ubuntu 18.04/20.04 on non-s390x
> +    - name: Install packages to build QEMU on Ubuntu 20.04 on non-s390x
>        package:
>          name:
>            - libspice-server-dev
> @@ -100,15 +99,6 @@
>          - ansible_facts['distribution'] == 'Ubuntu'
>          - ansible_facts['architecture'] != 's390x'
>  
> -    - name: Install basic packages to build QEMU on Ubuntu 18.04
> -      package:
> -        name:
> -        # Originally from tests/docker/dockerfiles/ubuntu1804.docker
> -          - clang
> -      when:
> -        - ansible_facts['distribution'] == 'Ubuntu'
> -        - ansible_facts['distribution_version'] == '18.04'
> -
>      - name: Install basic packages to build QEMU on Ubuntu 20.04
>        package:
>          name:
> diff --git a/tests/docker/dockerfiles/ubuntu1804.docker 
> b/tests/docker/dockerfiles/ubuntu1804.docker
> deleted file mode 100644
> index b3f2156580..0000000000
> --- a/tests/docker/dockerfiles/ubuntu1804.docker
> +++ /dev/null
> @@ -1,144 +0,0 @@
> -# THIS FILE WAS AUTO-GENERATED
> -#
> -#  $ lcitool dockerfile --layers all ubuntu-1804 qemu
> -#
> -# https://gitlab.com/libvirt/libvirt-ci
> -
> -FROM docker.io/library/ubuntu:18.04
> -
> -RUN export DEBIAN_FRONTEND=noninteractive && \
> -    apt-get update && \
> -    apt-get install -y eatmydata && \
> -    eatmydata apt-get dist-upgrade -y && \
> -    eatmydata apt-get install --no-install-recommends -y \
> -            bash \
> -            bc \
> -            bsdmainutils \
> -            bzip2 \
> -            ca-certificates \
> -            ccache \
> -            clang \
> -            dbus \
> -            debianutils \
> -            diffutils \
> -            exuberant-ctags \
> -            findutils \
> -            g++ \
> -            gcc \
> -            gcovr \
> -            genisoimage \
> -            gettext \
> -            git \
> -            glusterfs-common \
> -            hostname \
> -            libaio-dev \
> -            libasan5 \
> -            libasound2-dev \
> -            libattr1-dev \
> -            libbrlapi-dev \
> -            libbz2-dev \
> -            libc6-dev \
> -            libcacard-dev \
> -            libcap-ng-dev \
> -            libcapstone-dev \
> -            libcurl4-gnutls-dev \
> -            libdaxctl-dev \
> -            libdrm-dev \
> -            libepoxy-dev \
> -            libfdt-dev \
> -            libffi-dev \
> -            libgbm-dev \
> -            libgcrypt20-dev \
> -            libglib2.0-dev \
> -            libgnutls28-dev \
> -            libgtk-3-dev \
> -            libibumad-dev \
> -            libibverbs-dev \
> -            libiscsi-dev \
> -            libjemalloc-dev \
> -            libjpeg-turbo8-dev \
> -            liblttng-ust-dev \
> -            liblzo2-dev \
> -            libncursesw5-dev \
> -            libnfs-dev \
> -            libnuma-dev \
> -            libpam0g-dev \
> -            libpcre2-dev \
> -            libpixman-1-dev \
> -            libpmem-dev \
> -            libpng-dev \
> -            libpulse-dev \
> -            librbd-dev \
> -            librdmacm-dev \
> -            libsasl2-dev \
> -            libsdl2-dev \
> -            libsdl2-image-dev \
> -            libseccomp-dev \
> -            libselinux1-dev \
> -            libsnappy-dev \
> -            libspice-protocol-dev \
> -            libspice-server-dev \
> -            libssh-dev \
> -            libsystemd-dev \
> -            libtasn1-6-dev \
> -            libubsan1 \
> -            libudev-dev \
> -            libusb-1.0-0-dev \
> -            libusbredirhost-dev \
> -            libvdeplug-dev \
> -            libvirglrenderer-dev \
> -            libvte-2.91-dev \
> -            libxen-dev \
> -            libzstd-dev \
> -            llvm \
> -            locales \
> -            make \
> -            multipath-tools \
> -            netcat-openbsd \
> -            nettle-dev \
> -            ninja-build \
> -            openssh-client \
> -            perl-base \
> -            pkgconf \
> -            python3 \
> -            python3-numpy \
> -            python3-opencv \
> -            python3-pillow \
> -            python3-pip \
> -            python3-setuptools \
> -            python3-sphinx \
> -            python3-sphinx-rtd-theme \
> -            python3-venv \
> -            python3-wheel \
> -            python3-yaml \
> -            rpm2cpio \
> -            sed \
> -            sparse \
> -            systemtap-sdt-dev \
> -            tar \
> -            tesseract-ocr \
> -            tesseract-ocr-eng \
> -            texinfo \
> -            xfslibs-dev \
> -            zlib1g-dev && \
> -    eatmydata apt-get autoremove -y && \
> -    eatmydata apt-get autoclean -y && \
> -    sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \
> -    dpkg-reconfigure locales && \
> -    dpkg-query --showformat '${Package}_${Version}_${Architecture}\n' --show 
> > /packages.txt && \
> -    mkdir -p /usr/libexec/ccache-wrappers && \
> -    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/c++ && \
> -    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \
> -    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/clang && \
> -    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/g++ && \
> -    ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/gcc
> -
> -RUN pip3 install meson==0.56.0
> -
> -ENV LANG "en_US.UTF-8"
> -ENV MAKE "/usr/bin/make"
> -ENV NINJA "/usr/bin/ninja"
> -ENV PYTHON "/usr/bin/python3"
> -ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers"
> -# https://bugs.launchpad.net/qemu/+bug/1838763
> -ENV QEMU_CONFIGURE_OPTS --disable-libssh
> diff --git a/tests/lcitool/refresh b/tests/lcitool/refresh
> index 2d198ad281..fb49bbc441 100755
> --- a/tests/lcitool/refresh
> +++ b/tests/lcitool/refresh
> @@ -65,11 +65,6 @@ def generate_cirrus(target, trailer=None):
>     cmd = lcitool_cmd + ["variables", target, "qemu"]
>     generate(filename, cmd, trailer)
>  
> -ubuntu1804_skipssh = [
> -   "# https://bugs.launchpad.net/qemu/+bug/1838763\n";,
> -   "ENV QEMU_CONFIGURE_OPTS --disable-libssh\n"
> -]
> -
>  ubuntu2004_tsanhack = [
>     "# Apply patch https://reviews.llvm.org/D75820\n";,
>     "# This is required for TSan in clang-10 to compile with QEMU.\n",
> @@ -85,8 +80,6 @@ def debian_cross_build(prefix, targets):
>  try:
>     generate_dockerfile("centos8", "centos-stream-8")
>     generate_dockerfile("fedora", "fedora-35")
> -   generate_dockerfile("ubuntu1804", "ubuntu-1804",
> -                       trailer="".join(ubuntu1804_skipssh))
>     generate_dockerfile("ubuntu2004", "ubuntu-2004",
>                         trailer="".join(ubuntu2004_tsanhack))
>     generate_dockerfile("opensuse-leap", "opensuse-leap-152")
> diff --git a/tests/vm/ubuntu.aarch64 b/tests/vm/ubuntu.aarch64
> deleted file mode 100755
> index b291945a7e..0000000000
> --- a/tests/vm/ubuntu.aarch64
> +++ /dev/null
> @@ -1,68 +0,0 @@
> -#!/usr/bin/env python3
> -#
> -# Ubuntu aarch64 image
> -#
> -# Copyright 2020 Linaro
> -#
> -# Authors:
> -#  Robert Foley <robert.foley@linaro.org>
> -#  Originally based on ubuntu.i386 Fam Zheng <famz@redhat.com>
> -#
> -# This code is licensed under the GPL version 2 or later.  See
> -# the COPYING file in the top-level directory.
> -#
> -
> -import sys
> -import basevm
> -import aarch64vm
> -import ubuntuvm
> -
> -DEFAULT_CONFIG = {
> -    'cpu'          : "cortex-a57",
> -    'machine'      : "virt,gic-version=3",
> -    'install_cmds' : "apt-get update,"\
> -                     "apt-get build-dep -y --arch-only qemu,"\
> -                     "apt-get install -y libfdt-dev pkg-config 
> language-pack-en ninja-build",
> -    # We increase beyond the default time since during boot
> -    # it can take some time (many seconds) to log into the VM
> -    # especially using softmmu.
> -    'ssh_timeout'  : 60,
> -}
> -
> -class UbuntuAarch64VM(ubuntuvm.UbuntuVM):
> -    name = "ubuntu.aarch64"
> -    arch = "aarch64"
> -    image_name = "ubuntu-18.04-server-cloudimg-arm64.img"
> -    image_link = "https://cloud-images.ubuntu.com/releases/18.04/release/"; + 
> image_name
> -    
> image_sha256="0fdcba761965735a8a903d8b88df8e47f156f48715c00508e4315c506d7d3cb1"
> -    BUILD_SCRIPT = """
> -        set -e;
> -        cd $(mktemp -d);
> -        sudo chmod a+r /dev/vdb;
> -        tar --checkpoint=.10 -xf /dev/vdb;
> -        ./configure {configure_opts};
> -        make --output-sync {target} -j{jobs} {verbose};
> -    """
> -    def boot(self, img, extra_args=None):
> -        aarch64vm.create_flash_images(self._tmpdir, self._efi_aarch64)
> -        default_args = aarch64vm.get_pflash_args(self._tmpdir)
> -        if extra_args:
> -            extra_args.extend(default_args)
> -        else:
> -            extra_args = default_args
> -        # We always add these performance tweaks
> -        # because without them, we boot so slowly that we
> -        # can time out finding the boot efi device.
> -        if '-smp' not in extra_args and \
> -           '-smp' not in self._config['extra_args'] and \
> -           '-smp' not in self._args:
> -            # Only add if not already there to give caller option to change 
> it.
> -            extra_args.extend(["-smp", "8"])
> -
> -        # We have overridden boot() since aarch64 has additional parameters.
> -        # Call down to the base class method.
> -        super(UbuntuAarch64VM, self).boot(img, extra_args=extra_args)
> -
> -if __name__ == "__main__":
> -    defaults = aarch64vm.get_config_defaults(UbuntuAarch64VM, DEFAULT_CONFIG)
> -    sys.exit(basevm.main(UbuntuAarch64VM, defaults))
> diff --git a/tests/vm/ubuntu.i386 b/tests/vm/ubuntu.i386
> deleted file mode 100755
> index 47681b6f87..0000000000
> --- a/tests/vm/ubuntu.i386
> +++ /dev/null
> @@ -1,40 +0,0 @@
> -#!/usr/bin/env python3
> -#
> -# Ubuntu i386 image
> -#
> -# Copyright 2017 Red Hat Inc.
> -#
> -# Authors:
> -#  Fam Zheng <famz@redhat.com>
> -#
> -# This code is licensed under the GPL version 2 or later.  See
> -# the COPYING file in the top-level directory.
> -#
> -
> -import sys
> -import basevm
> -import ubuntuvm
> -
> -DEFAULT_CONFIG = {
> -    'install_cmds' : "apt-get update,"\
> -                     "apt-get build-dep -y qemu,"\
> -                     "apt-get install -y libfdt-dev language-pack-en 
> ninja-build",
> -}
> -
> -class UbuntuX86VM(ubuntuvm.UbuntuVM):
> -    name = "ubuntu.i386"
> -    arch = "i386"
> -    image_link="https://cloud-images.ubuntu.com/releases/bionic/"\
> -               "release-20191114/ubuntu-18.04-server-cloudimg-i386.img"
> -    
> image_sha256="28969840626d1ea80bb249c08eef1a4533e8904aa51a327b40f37ac4b4ff04ef"
> -    BUILD_SCRIPT = """
> -        set -e;
> -        cd $(mktemp -d);
> -        sudo chmod a+r /dev/vdb;
> -        tar -xf /dev/vdb;
> -        ./configure {configure_opts};
> -        make --output-sync {target} -j{jobs} {verbose};
> -    """
> -
> -if __name__ == "__main__":
> -    sys.exit(basevm.main(UbuntuX86VM, DEFAULT_CONFIG))
> diff --git a/tests/vm/ubuntuvm.py b/tests/vm/ubuntuvm.py
> deleted file mode 100644
> index 6689ad87aa..0000000000
> --- a/tests/vm/ubuntuvm.py
> +++ /dev/null
> @@ -1,60 +0,0 @@
> -#!/usr/bin/env python3
> -#
> -# Ubuntu VM testing library
> -#
> -# Copyright 2017 Red Hat Inc.
> -# Copyright 2020 Linaro
> -#
> -# Authors:
> -#  Robert Foley <robert.foley@linaro.org>
> -#  Originally based on ubuntu.i386 Fam Zheng <famz@redhat.com>
> -#
> -# This code is licensed under the GPL version 2 or later.  See
> -# the COPYING file in the top-level directory.
> -
> -import os
> -import subprocess
> -import basevm
> -
> -class UbuntuVM(basevm.BaseVM):
> -
> -    def __init__(self, args, config=None):
> -        self.login_prompt = "ubuntu-{}-guest login:".format(self.arch)
> -        basevm.BaseVM.__init__(self, args, config)
> -
> -    def build_image(self, img):
> -        """Build an Ubuntu VM image.  The child class will
> -           define the install_cmds to init the VM."""
> -        os_img = self._download_with_cache(self.image_link,
> -                                           sha256sum=self.image_sha256)
> -        img_tmp = img + ".tmp"
> -        subprocess.check_call(["cp", "-f", os_img, img_tmp])
> -        self.exec_qemu_img("resize", img_tmp, "+50G")
> -        ci_img = self.gen_cloud_init_iso()
> -
> -        self.boot(img_tmp, extra_args = [ "-device", "VGA", "-cdrom", 
> ci_img, ])
> -
> -        # First command we issue is fix for slow ssh login.
> -        self.wait_ssh(wait_root=True,
> -                      cmd="chmod -x /etc/update-motd.d/*")
> -        # Wait for cloud init to finish
> -        self.wait_ssh(wait_root=True,
> -                      cmd="ls /var/lib/cloud/instance/boot-finished")
> -        self.ssh_root("touch /etc/cloud/cloud-init.disabled")
> -        # Disable auto upgrades.
> -        # We want to keep the VM system state stable.
> -        self.ssh_root('sed -ie \'s/"1"/"0"/g\' '\
> -                      '/etc/apt/apt.conf.d/20auto-upgrades')
> -        self.ssh_root("sed -ie s/^#\ deb-src/deb-src/g 
> /etc/apt/sources.list")
> -
> -        # If the user chooses not to do the install phase,
> -        # then we will jump right to the graceful shutdown
> -        if self._config['install_cmds'] != "":
> -            # Issue the install commands.
> -            # This can be overriden by the user in the config .yml.
> -            install_cmds = self._config['install_cmds'].split(',')
> -            for cmd in install_cmds:
> -                self.ssh_root(cmd)
> -        self.graceful_shutdown()
> -        os.rename(img_tmp, img)
> -        return 0
> -- 
> 2.27.0
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|