Re: [PATCH] bootstrap: When a commit hash is specified, do a shallow fetch if possible

[Glenn Washburn]

On Thu, 21 Oct 2021 13:31:31 -0500
Glenn Washburn <development@efficientek.com> wrote:

> The gnulib sources are large but more importantly have lots of changes. So
> initial checkout of the repository can take a long time when network or
> cpu resources are limited. The later is especially acute in a non-KVM QEMU
> virtual machine (which can take 40+ minutes compared to <30 seconds with
> this change[1]). The problem is specific to bootstrap configurations using
> a specific gnulib revision specified by commit hash. In this case, git can
> not do a shallow clone by using the --depth option because git does not know
> ahead of time how deep the revision is from the tip. So git must clone the
> whole repository.
> 
> However, there is an alternate method that requires support from the git
> server[2], namely by asking for a specific commit on fetch. Refactor to use
> fetch and fallback to fetching the entire repository if fetching by commit
> hash fails.
> 
> Currently the git server hosting the official gnulib git repository does not
> support fetch by commit hash[3]. However, there are mirrors which do support
> this[4], and can be specified by setting the $GNULIB_URL.

The issue with the savannah git server got resolved more quickly than
anticipated. I suggest cutting out the last paragraph and the last two
links below. No need to mention the non-Free service. I can recreate
the patch if that's desired as well, but have no issue with a
maintainer editing the commit message before committing.

Glenn

> 
> [1] https://savannah.nongnu.org/support/index.php?110553#comment1
> [2] https://stackoverflow.com/a/3489576/2108011
> [3] https://savannah.nongnu.org/support/index.php?110553
> [4] https://github.com/coreutils/gnulib
> 
> Signed-off-by: Glenn Washburn <development@efficientek.com>
> ---
> Hi list,
> 
> This patch can have a dramatic performance improvement in checking out gnulib
> sources via bootstrap. These changes use the git -C which was introduced in
> 2013 in v1.8.5, so I think its been around long enough to rely on it. Also, 
> I'm
> not subscribed to the list, so please ensure to CC me on any replies to this
> patch.
> 
> Thanks,
> Glenn
> 
> ---
>  build-aux/bootstrap | 16 ++++++++++++++--
>  1 file changed, 14 insertions(+), 2 deletions(-)
> 
> diff --git a/build-aux/bootstrap b/build-aux/bootstrap
> index 733659850..9f8b918a2 100755
> --- a/build-aux/bootstrap
> +++ b/build-aux/bootstrap
> @@ -694,9 +694,21 @@ if $use_gnulib; then
>        shallow=
>        if test -z "$GNULIB_REVISION"; then
>          git clone -h 2>&1 | grep -- --depth > /dev/null && shallow='--depth 
> 2'
> +        git clone $shallow ${GNULIB_URL:-$default_gnulib_url} "$gnulib_path" 
> \
> +          || cleanup_gnulib
> +      else
> +        git fetch -h 2>&1 | grep -- --depth > /dev/null && shallow='--depth 
> 2'
> +        mkdir -p "$gnulib_path"
> +        git -C "$gnulib_path" init
> +        git -C "$gnulib_path" remote add origin 
> ${GNULIB_URL:-$default_gnulib_url}
> +        # Can not do a shallow fetch if fetch by commit hash fails because we
> +        # do not know how deep to go to get to $GNULIB_REVISION, so we must 
> get
> +        # all commits.
> +        git -C "$gnulib_path" fetch $shallow origin "$GNULIB_REVISION" \
> +          || git -C "$gnulib_path" fetch origin \
> +          || cleanup_gnulib
> +        git -C "$gnulib_path" reset --hard FETCH_HEAD
>        fi
> -      git clone $shallow ${GNULIB_URL:-$default_gnulib_url} "$gnulib_path" \
> -        || cleanup_gnulib
>  
>        trap - 1 2 13 15
>      fi