v1.1.0 (~2020) and SWH: Part II

[Simon Tournier]

Hi,

Let continue the investigations.  Now, assume that:

    guix time-machine -C /tmp/v1.1.0cm -- help

passes, where /tmp/v1.1.0.scm is defined elsewhere in this thread.

We spot these issues:

 + Holes in the coverage

        guile-json-3.2.0.tar.gz.drv
        net-tools-1.60-0.479bb4a.zip.drv
        hidapi-0.9.0-checkout.drv

 + Failures to re-build

        python-minimal-3.7.4.drv
        gnutls-3.6.A.drv

 + Hash mismatch (hum?!)

   + Probably from libgit2 not dealing correctly with crlf and .gitattribute,
     Need more investigations…

 + Missing features in (guix swh)

   + lightweight tags
   + zip archive (Disarchive side :-))

 + SWH request rate limits
   something to discuss for “Reproducible Science” at scale. :-)


To be continued…

Cheers,
simon

- - -

(Again, it appears to me weird to have to specify --no-substitutes where
the substitutes servers are unreachable.  Well, maybe it had been
corrected since this old version v1.1.0. :-))

# Preliminary

Using only,

     archive.softwareheritage.org
     disarchive.guix.gnu.org

then

    guix time-machine -C /tmp/v1.1.0.scm -- build hello --no-substitutes

fails because many missing dependencies.  I will not list all since, let
say, it is part of the issues pointed by Part I.  As said previously,
the addition of git.savannah.gnu.org helps which confirms that we have
holes with our coverage of SWH + Disarchive.  Then, we hit ed and all so
let add ftp.gnu.org for building hello.  Missing,

/gnu/store/qwbmqzyqv8nl39pkmzyp268lcnjrhrvs-linux-libre-4.19.56-gnu.tar.xz.drv
/gnu/store/85p96nq7i86cav03m5ahxlcldzd8wq8c-nyacc-0.86.0.tar.gz.drv
/gnu/store/sjf06i58f3bbhfggw6sj3n742zkn9632-static-binaries.tar.xz.drv

And note that
https://ftp.gnu.org/gnu/guix/bootstrap/x86_64-linux/20131110/static-binaries.tar.xz
is gone.

Well, let enable the network and run:

    guix time-machine -C /tmp/v1.1.0.scm -- build hello
    guix time-machine -C /tmp/v1.1.0.scm -- build hello --check --no-grafts

That’s said, let do some tests about the coverage.  Without specific
mention, we consider only this:

    archive.softwareheritage.org
    disarchive.guix.gnu.org
    git.savannah.gnu.org
    ftp.gnu.org


# Github is down

Let build open-zwave which reads,

--8<---------------cut here---------------start------------->8---
              (uri (git-reference
                    (url "https://github.com/OpenZWave/open-zwave/";)
                    (commit (string-append "v" version))))
--8<---------------cut here---------------end--------------->8---

using,

    guix time-machine -C /tmp/v1.1.0.scm -- build open-zwave --no-substitutes

and it fails with,

/gnu/store/mvmi5fsv42zv6dn4g5w7r3mw8mqrgzik-guile-json-3.2.0.tar.gz.drv
/gnu/store/9ry22bncalynd5is3icrlaib4570gya0-net-tools-1.60-0.479bb4a.zip.drv
/gnu/store/2s25w9k5bi3hkr5sgmvn1d4hl886bchp-hidapi-0.9.0-checkout.drv

/gnu/store/984hkbdjp9dskn9yxwha61q9i6nbl11v-python-minimal-3.7.4.drv    <-- 
annoying failures!
/gnu/store/sshwk1xqaf1c1ms3cm93brzaizk15c7l-python-3.7.4.drv            <--
/gnu/store/qm3l79ic89qpjjd8avqxd81425v4wvv5-gnutls-3.6.A.drv            <--

And we hit a bug!

--8<---------------cut here---------------start------------->8---
fatal: unable to access 'https://github.com/libusb/hidapi.git/': Could not 
resolve host: github.com
Failed to do a shallow fetch; retrying a full fetch...
fatal: unable to access 'https://github.com/libusb/hidapi.git/': Could not 
resolve host: github.com
git-fetch: 
'/gnu/store/xzhkhidb2kqwvpv8b8zaqzgd3gcjs9nn-git-minimal-2.26.0/bin/git fetch 
origin' failed with exit code 128
Trying content-addressed mirror at berlin.guixsd.org...
Trying content-addressed mirror at berlin.guixsd.org...
Trying to download from Software Heritage...
SWH: found revision 7da5cc91fc0d2dbe4df4f08cd31f6ca1a262418f with directory at 
'https://archive.softwareheritage.org/api/1/directory/c95078f3645abc050536d3aaf76cb5fb6eb6b288/'
SWH vault: requested bundle cooking, waiting for completion...
swh:1:dir:c95078f3645abc050536d3aaf76cb5fb6eb6b288/

[...]

swh:1:dir:c95078f3645abc050536d3aaf76cb5fb6eb6b288/windows/hidtest.vcproj
r:sha256 hash mismatch for 
/gnu/store/n6rvypb3z386v0ml6n0xyga2x4q9ij58-hidapi-0.9.0-checkout:
  expected hash: 1p4g8lgwj4rki6lbn5l6rvwj0xlbn1xfh4d255bg5pvgczmwmc4i
  actual hash:   1cbbg4rkfpjwkgqva2659zasqr5d5rz6bi71malnnzy0y2qin2pz
hash mismatch for store item 
'/gnu/store/n6rvypb3z386v0ml6n0xyga2x4q9ij58-hidapi-0.9.0-checkout'
--8<---------------cut here---------------end--------------->8---

Quoting, vlorentz from SWH:

        <vlorentz> zimoun: so in short, the discrepency is probably because 
your git
                   doesn't honor the crlf setting in .gitattribute but used to, 
or
                   vice versa  

So probably libgit2 incorrectly dealing with .gitattribute from hidapi
repository.

Last, another more annoying one:

--8<---------------cut here---------------start------------->8---
Initialized empty Git repository in 
/gnu/store/zlmgdc8c9mgb9l17fchkarsn7hf3s39r-open-zwave-1.6-checkout/.git/
fatal: unable to access 'https://github.com/OpenZWave/open-zwave/': Could not 
resolve host: github.com
Failed to do a shallow fetch; retrying a full fetch...
fatal: unable to access 'https://github.com/OpenZWave/open-zwave/': Could not 
resolve host: github.com
git-fetch: 
'/gnu/store/xzhkhidb2kqwvpv8b8zaqzgd3gcjs9nn-git-minimal-2.26.0/bin/git fetch 
origin' failed with exit code 128
Trying content-addressed mirror at berlin.guixsd.org...
Trying content-addressed mirror at berlin.guixsd.org...
Trying to download from Software Heritage...
builder for 
`/gnu/store/bcdsz16z454g756fsnrqrs92r7z1dqwa-open-zwave-1.6-checkout.drv' 
failed to produce output path 
`/gnu/store/zlmgdc8c9mgb9l17fchkarsn7hf3s39r-open-zwave-1.6-checkout'
build of 
/gnu/store/bcdsz16z454g756fsnrqrs92r7z1dqwa-open-zwave-1.6-checkout.drv failed
--8<---------------cut here---------------end--------------->8---

Well, this repository is saved in SWH and the release tag v1.6 is
ingested, as shown here:

https://archive.softwareheritage.org/browse/snapshot/93d83c96be2bbb2ea9a172c3fdacb80413bc276c/branches/?origin_url=https://github.com/OpenZWave/open-zwave

but, quoting olasd from SWH:

        <olasd> zimoun: it is; but it's a lightweight tag so it only shows in 
the
                "branches" list

It means we have some missing feature on our side.

Ok!  Now, let check for all the packages with github.com as upstream,
using this manifest file:

--8<---------------cut here---------------start------------->8---
(use-modules (guix) (gnu)
             (guix git-download)
             (ice-9 match)
             (srfi srfi-1)
             (srfi srfi-26))

(define (that-url? package)
  (define (that-string? str)
    (string-contains str "github.com"))

  (match (package-source package)
    ((? origin? o)
     (match (origin-uri o)
       ((? string? url)
        (that-string? url))
       (((? string? urls) ...)
        (any that-string? urls))

       ((? git-reference? ref)
        (that-string? (git-reference-url ref)))
       (_ #f)))
    (_ #f)))


(manifest
 (map package->manifest-entry
      (fold-packages (lambda (package result)
                       (if (that-url? package)
                           (cons package result)
                           result))
                     '())))
--8<---------------cut here---------------end--------------->8---

and,

    guix time-machine -C /tmp/v1.1.0.scm \
         -- build -m /tmp/from-that-url.scm -S --no-substitutes -k

fails at some points.  One is because we hit the SWH rate limit.  Other
are, among many others,

/gnu/store/867xgw7syxkm32ryjcnrz7cxc1gklz4a-android-googletest-1.8.0-checkout.drv
/gnu/store/hrzhpb2xp682ikcgrnrl5y2651hr9wn6-android-make-stub-0.6.0-checkout.drv
/gnu/store/43mkyklyn9sqz3wnhlrp313xcbm57j0p-apache-arrow-0.10.0-checkout.drv
/gnu/store/5ybwii43g0nclcx527y2ws8s6ai81rsj-apfs-fuse-0.0.0-0.c7036a3-checkout.drv
/gnu/store/66xzcip9rbsig3y5dbyz5xyyaqzszpyp-arc-icon-theme-20161122-checkout.drv
/gnu/store/gyvhlabn6xn6zav76cpx7xg82kfisniq-arducopter-bbbmini-3.6.11-checkout.drv
/gnu/store/y8rajri7kykv1yh8ykk4a3ighswjiadj-arducopter-bebop-3.6.11-checkout.drv
/gnu/store/wg12wwb2yb6nzmg1ani6jm2dnmi7r8n5-arduplane-bbbmini-4.0.1-checkout.drv
[...]
/gnu/store/64zx0dviwm3jbkqaka3nrri1azgw9bsh-allegro-4.4.3.tar.gz.drv
/gnu/store/5l1g6n291s2m7fn4hypk5q66vh9nw51v-alot-0.5.1.tar.gz.drv
/gnu/store/x9mr8dkn4rirr8z3sqq9nfkh0gx5mdp0-apertium-3.5.2.tar.gz.drv
[...]
/gnu/store/p2izqfpw3k1jvq20m43f3bn5mpj8fxh9-apsw-3.28.0-r1.zip.drv

To be continued… for instance gitlab.com, bioconductor.org, etc.