Re: Disarchive database synchronization

[Timothy Sample]

Hey Ludo,

Ludovic Courtès <ludovic.courtes@inria.fr> writes:

> I copied over the 12K entries that were missing from
> disarchive.guix.gnu.org.  (Note that there are currently only two copies
> of the database: one at/in [bB]erlin, and one at/in [Bb]ordeaux.)
> disarchive.guix.gnu.org now weighs in at 1.8 GiB for 31,839 entries.

Wow – 12K!  For some reason I thought it would be fewer.  It’s very good
that we (finally) sync’d up the databases.

Also, my set is now at 31,821 after collecting the runoff from the
latest Preservation of Guix Report.  That’s shockingly close to the
31,839 you have.

> For the remaining entries, it’s trickier.  Sometimes it’s just the
> gzip compression parameters that differ, which could be addressed with a
> little bit more work:
>
> $ file ffdc77f5e5cb2390b9309de63eb7be68d9fe631e898f4da6c04a8159daefc2c0.gz 
> ../../disarchive/sha256/ffdc77f5e5cb2390b9309de63eb7be68d9fe631e898f4da6c04a8159daefc2c0.gz
> ffdc77f5e5cb2390b9309de63eb7be68d9fe631e898f4da6c04a8159daefc2c0.gz:          
>                gzip compressed data, max compression, from Unix, original 
> size modulo 2^32 446731
> ../../disarchive/sha256/ffdc77f5e5cb2390b9309de63eb7be68d9fe631e898f4da6c04a8159daefc2c0.gz:
>  gzip compressed data, max speed, from Unix, original size modulo 2^32 446731

I’m not sure getting the compressed files to match matters.  Disarchive
cares a lot about that when it comes to source code tarballs, because
everybody signs and computes checksums over the compressed versions.
However, for these files, the differences introduced by compression can
be ignored.

> Sometimes it’s trickier:
>
> # diff -u <(gunzip -d < 
> 0001f025c1425ffe36270a81cb091eade87dd8d29ac773735ae47e1a8c8066c9.gz) <(gunzip 
> -d < 
> ../../disarchive/sha256/0001f025c1425ffe36270a81cb091eade87dd8d29ac773735ae47e1a8c8066c9.gz)
> --- /dev/fd/63  2023-03-14 16:13:21.635733426 +0100
> +++ /dev/fd/62  2023-03-14 16:13:21.635733426 +0100
> @@ -1,7 +1,7 @@
>  (disarchive
>    (version 0)
>    (gzip-member
> -    (name "webview-sys-0.6.2.tar.gz")
> +    (name "rust-webview-sys-0.6.2.tar.gz")
>      (digest
>        (sha256
>          "0001f025c1425ffe36270a81cb091eade87dd8d29ac773735ae47e1a8c8066c9"))
> @@ -13,7 +13,7 @@
>      (footer (crc 1807070134) (isize 121344))
>      (compressor zlib-best)
>      (input (tarball
> -             (name "webview-sys-0.6.2.tar")
> +             (name "rust-webview-sys-0.6.2.tar")
>               (digest
>                 (sha256
>                   
> "4fb18f3206838e11f7f8caba6fad9e0f796109428b502793b9f2f0613fe0f275"))
> @@ -78,7 +78,7 @@
>               (padding 0)
>               (input (directory-ref
>                        (version 0)
> -                      (name "webview-sys-0.6.2")
> +                      (name "rust-webview-sys-0.6.2")
>                        (addresses
>                          (swhid 
> "swh:1:dir:fa41df38bf639ada28c900b0915661e787fe6d15"))
>                        (digest

The name field is not used for data reconstruction.  It’s for human
consumption (and it may have made some early examples of use at the
command line easier to explain).  Here, the difference is based on the
fact that Crate URIs are weird, and the Preservation of Guix code does
not keep the origin file name.  Hence, the PoG version extracts the
Crate name alone from the URI, and the Cuirass version uses the Guix
package name with the “rust-” prefix.

> As Tim pointed out, Disarchive disassembly is not fully deterministic
> and/or might change a bit over time as Disarchive evolves, and that’s
> prolly what we’re seeing here.

I honestly think this is a good thing.  My instincts tell me that we
should excise all sources of ambiguity, like we’re trying to do in the
big picture.  However, Disarchive will get better at describing things
over time.  For instance, it doesn’t handle tar extension headers
elegantly at the moment.  In the future, if I fix this, I might consider
creating a “migrate” feature that improves existing specifications
(e.g., converting the old, verbose representation of extension headers
into the new representation).  In particular, I’ve left some warts in
the software in order to ship it, and I would be sad to try and commit
to those for the rest of time!

We might also add other resolver addresses besides SWHIDs....

Maybe I’m missing some perspective, but I don’t think trying to commit
to reproducible outputs for Disarchive makes sense.


-- Tim

P.S., we’ll have to do this dance again shortly, as I just computed
2,023 historical bzip2 specifications.  They’re not online yet, but
they’ll be up when I publish the next PoG report – which should take less
than a year this time!  :p