Re: [PATCH 11/11] tests/gitlab: use kaniko to build images

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Thu, Mar 30, 2023 at 11:17:41AM +0100, Daniel P. Berrangé wrote:
>> On Thu, Mar 30, 2023 at 11:11:41AM +0100, Alex Bennée wrote:
>> > Apparently the docker-in-docker approach has some flaws including
>> > needing privileged mode to run and being quite slow. An alternative
>> > approach is to use Google's kaniko tool. It also works across
>> > different gitlab executors.
>> > 
>> > Following the gitlab example code we drop all the direct docker calls
>> > and usage of the script and make a direct call to kaniko and hope the
>> > images are cacheable by others.
>> > 
>> > Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> > Message-Id: <20230224180857.1050220-8-alex.bennee@linaro.org>
>> > 
>> > ---
>> > v2
>> >   - add danpb's --cache suggestions
>> > ---
>> >  .gitlab-ci.d/container-template.yml | 22 ++++++++++------------
>> >  1 file changed, 10 insertions(+), 12 deletions(-)
>> > 
>> > diff --git a/.gitlab-ci.d/container-template.yml 
>> > b/.gitlab-ci.d/container-template.yml
>> > index 519b8a9482..cd8e0a1ff6 100644
>> > --- a/.gitlab-ci.d/container-template.yml
>> > +++ b/.gitlab-ci.d/container-template.yml
>> > @@ -1,21 +1,19 @@
>> >  .container_job_template:
>> >    extends: .base_job_template
>> > -  image: docker:stable
>> > +  image:
>> > +    name: gcr.io/kaniko-project/executor:v1.9.0-debug
>> > +    entrypoint: [""]
>> >    stage: containers
>> > -  services:
>> > -    - docker:dind
>> >    before_script:
>> >      - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
>> >      - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
>> > -    - apk add python3
>> > -    - docker info
>> > -    - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p 
>> > "$CI_REGISTRY_PASSWORD"
>> >    script:
>> >      - echo "TAG:$TAG"
>> >      - echo "COMMON_TAG:$COMMON_TAG"
>> > -    - docker build --tag "$TAG" --cache-from "$TAG" --cache-from 
>> > "$COMMON_TAG"
>> > -      --build-arg BUILDKIT_INLINE_CACHE=1
>> > -      -f "tests/docker/dockerfiles/$NAME.docker" "."
>> > -    - docker push "$TAG"
>> > -  after_script:
>> > -    - docker logout
>> > +    - /kaniko/executor
>> > +          --reproducible
>> > +          --context "${CI_PROJECT_DIR}"
>> > +          --cache=true
>> > +          --cache-repo "${COMMON_TAG}"
>> 
>> IIRC with docker if we told it to cache we would have to first have done
>> a  'docker pull $COMMON_TAG' as it wouldn't pull down the image if
>> it was not already local. I'm fuzzy on whether kaniko has the same
>> need or not ?  I guess we were broken already in that respect as
>> we already uses --cache-from with docker without a docker pull
>
> Oh never mind, because we're not docker-in-docker, we can't pull the
> image tag down locally, and as discussed on IRC, caching works in a
> very different way. kaniko wants to be able to push & pull in the
> cache-repo itself.
>
> I'm inclined to think we're better off ignoring layer caching and instead
> focus on entirely skipping execution of kaniko if we know the dockerfile
> has not changed eg something along the lines of:
>
>    manifest=$(curl ....some registry URL to fetch image metadata)
>    oldchecksum=$(...extract a LABEL from metadata container dockerfile sha256)
>    newchecksum=$(sha256sum tests/docker/dockerfiles/$NAME.docker)
>
>    if test $oldchecksum != $newchecksum -o -n $QEMU_FORCE_REBUILD"
>    then
>       - /kaniko/executor
>             --reproducible
>             --context "${CI_PROJECT_DIR}"
>             --dockerfile 
> "${CI_PROJECT_DIR}/tests/docker/dockerfiles/$NAME.docker"
>           --label DKR_CHECKSUM=$newchecksum
>             --destination "${TAG}"
>    fi
>
>
> And then have a weekly pipeline on sundays that sets QEMU_FORCE_REBUILD=1
> so that we pick up changes from the distro base images, and/or package
> repes regularly.

Hmm this appears to be a dead end. I got to this:

--8<---------------cut here---------------start------------->8---
tests/gitlab: use kaniko to build images

Apparently the docker-in-docker approach has some flaws including
needing privileged mode to run and being quite slow. An alternative
approach is to use Google's kaniko tool. It also works across
different gitlab executors.

Following the gitlab example code we drop all the direct docker calls
and usage of the script and make a direct call to kaniko and hope the
images are cacheable by others.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230224180857.1050220-8-alex.bennee@linaro.org>

---
v2
  - add danpb's --cache suggestions
v3
  - don't include :latest in tag
  - allow kaniko to infer local registry location, drop COMMON_TAG
  - add registry login details
  - version bump
  - don't push cache layers

1 file changed, 13 insertions(+), 14 deletions(-)
.gitlab-ci.d/container-template.yml | 27 +++++++++++++--------------

modified   .gitlab-ci.d/container-template.yml
@@ -1,21 +1,20 @@
 .container_job_template:
   extends: .base_job_template
-  image: docker:stable
+  image:
+    name: gcr.io/kaniko-project/executor:v1.9.2-debug
+    entrypoint: [""]
   stage: containers
-  services:
-    - docker:dind
   before_script:
-    - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest"
-    - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest"
-    - apk add python3
-    - docker info
-    - docker login $CI_REGISTRY -u "$CI_REGISTRY_USER" -p 
"$CI_REGISTRY_PASSWORD"
+    - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME"
   script:
     - echo "TAG:$TAG"
     - echo "COMMON_TAG:$COMMON_TAG"
-    - docker build --tag "$TAG" --cache-from "$TAG" --cache-from "$COMMON_TAG"
-      --build-arg BUILDKIT_INLINE_CACHE=1
-      -f "tests/docker/dockerfiles/$NAME.docker" "."
-    - docker push "$TAG"
-  after_script:
-    - docker logout
+    - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(echo -n 
${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64)\"}}}" > 
/kaniko/.docker/config.json
+    - /kaniko/executor
+          --reproducible
+          --context "${CI_PROJECT_DIR}"
+          --cache=true
+          --reproducible
+          --no-push-cache
+          --dockerfile 
"${CI_PROJECT_DIR}/tests/docker/dockerfiles/$NAME.docker"
+          --destination "${TAG}"
--8<---------------cut here---------------end--------------->8---

However the builds are failing so I think I just need to drop this and
move on.

>
> With regards,
> Daniel


-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro