Re: [RFC PATCH] ci: ensure that all jobs use a shallow clone

[Daniel P . Berrangé]

On Thu, Jan 07, 2021 at 06:17:19PM +0100, Paolo Bonzini wrote:
> We are still using quite a bit of bandwidth to run CI jobs, even though
> GitLab has switched to gitlab.com to fetch the sources.  This is in
> part because we are handling submodules ourselves and therefore those
> do not use shallow clones.

I presume the impact is going to vary for GitLab vs the other
CI systems.  For gitlab, any bandwidth used should be entirely
within gitlab's own network infra since its pulling repos
from  gitlab.com/qemu-project

For Cirrus/Shippable/Travis  the bandwidth used would be more
of a potential performance problem.

> Observe GitLab's GIT_DEPTH environment variable in
> scripts/git-submodule.sh so that CI systems can use a shallow clone
> for submodules.  Set a depth of 3 for all CI systems, for both the main
> repository (using whatever native configuration mechanism is there)
> and the submodules.
> 
> Reported-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>         Completely untested, just to propose the idea.

The check-patch and check-dco jobs need the git history depth to be
long enough to cover the entire contents of the branch that is being
tested vs common ancestor against git master.

This could potentially be 100's of commits deep for some of the very
largest branches we see merged in qemu


Travis has issues with git cloning and concurrent pushes.

eg if you push branch A, it schedules a CI job. Then you
push branch B before jobs for A have started.

When the job for A starts, it will be unable to checkout
the commit for branch A's merge because the history will
be too shallow.


I'm not sure if the other CI systems have similar issues or
not.

> 
>  .cirrus.yml              | 1 +
>  .gitlab-ci.yml           | 3 +++
>  .shippable.yml           | 2 ++
>  .travis.yml              | 2 ++
>  scripts/git-submodule.sh | 2 +-
>  5 files changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/.cirrus.yml b/.cirrus.yml
> index 6f2a958472..47c2467d04 100644
> --- a/.cirrus.yml
> +++ b/.cirrus.yml
> @@ -1,5 +1,6 @@
>  env:
> -  CIRRUS_CLONE_DEPTH: 1
> +  CIRRUS_CLONE_DEPTH: 3
> +  GIT_DEPTH: 3
>  
>  freebsd_12_task:
>    freebsd_instance:
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 01c9e46410..ff19a9c313 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -7,6 +7,9 @@ stages:
>    - build
>    - test
>  
> +variables:
> +  GIT_DEPTH: 3
> +
>  include:
>    - local: '/.gitlab-ci.d/edk2.yml'
>    - local: '/.gitlab-ci.d/opensbi.yml'
> diff --git a/.shippable.yml b/.shippable.yml
> index 14350e6de8..f2d59b478e 100644
> --- a/.shippable.yml
> +++ b/.shippable.yml
> @@ -1,9 +1,11 @@
>  language: c
> +depth: 3
>  git:
>     submodules: false
>  env:
>    global:
>      - LC_ALL=C
> +    - GIT_DEPTH=3
>    matrix:
>      - IMAGE=debian-amd64
>        TARGET_LIST=x86_64-softmmu,x86_64-linux-user
> diff --git a/.travis.yml b/.travis.yml
> index f2a101936c..f2a785a42b 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -80,9 +80,11 @@ env:
>      - CCACHE_SLOPPINESS="include_file_ctime,include_file_mtime"
>      - CCACHE_MAXSIZE=1G
>      - G_MESSAGES_DEBUG=error
> +    - GIT_DEPTH=3
>  
>  
>  git:
> +  depth: 3
>    # we want to do this ourselves
>    submodules: false
>  
> diff --git a/scripts/git-submodule.sh b/scripts/git-submodule.sh
> index 65ed877aef..2886087a12 100755
> --- a/scripts/git-submodule.sh
> +++ b/scripts/git-submodule.sh
> @@ -75,7 +75,7 @@ update)
>          exit 0
>      fi
>  
> -    $GIT submodule update --init $modules 1>/dev/null
> +    $GIT submodule update ${GIT_DEPTH:+--depth "$GIT_DEPTH"} --init $modules 
> 1>/dev/null
>      test $? -ne 0 && error "failed to update modules"
>  
>      $GIT submodule status $modules > "${substat}"
> -- 
> 2.29.2
> 
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|