Re: Non deterministic package

[Efraim Flashner]

I've become involved in the rust packaging effort so I'll jump in with
some specifics.

On Wed, Aug 04, 2021 at 03:50:00PM -0400, Julien Lepiller wrote:
> 
> 
> Le 4 août 2021 15:14:16 GMT-04:00, phodina via <help-guix@gnu.org> a écrit :
> >Hi,
> >
> >I came across a package in the Guix repository which fails to build. The 
> >package in question here is rust-brotli-sys-0.3. So the obvious question 
> >comes up. How does this happen?
> >
> >I just ran =guix build rust-brotli-sys= with the following output:
> >
> >...
> >phase `reset-gzip-timestamps' succeeded after 0.0 seconds
> >starting phase `compress-documentation'
> >phase `compress-documentation' succeeded after 0.0 seconds
> >guix build: error: derivation 
> >`/gnu/store/kf8kwl7yn8jqpwc9p97bw8vs5g5bvas4-rust-brotli-sys-0.3.2.drv' may 
> >not be deterministic: output 
> >`/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2' differs
> >Also this is my guix channel:
> >
> >guix:
> >repository URL: https://git.savannah.gnu.org/git/guix.git
> >branch: master
> >commit: b95fb85f593d377864fc1bfb3c118e54d03783be
> >
> >I tried to run the build with the flag --check and multiple rounds, but got 
> >the same result.
> >So could it be caused by not running =./pre-inst-env guix build --rounds=2 
> ><PKG>= when submitting the patch?
> 
> You need to use. /pre-inst-env to test your patches, otherwise you'll only 
> run your current guix. For an added package, it will be missing, for an 
> updated package it will not build the newer version, etc.
> 
> >
> >Or when somebody updates some pkg that this pkg depends on and does not run 
> >=./pre-inst-env guix --refresh --list-dependent= and then check the 
> >dependent pkgs?
> 
> It can also happen when only the current day is recorded in the output: if 
> you use —rounds=2, it is likely run on the same day, so the output will be 
> the same (don't bother though, last time I found such an issue it was in 
> january because only the year was recorded ^^")
> 
> You should now use -K to keep the different version and run diffoscope to 
> analyze the differences. Then we should be able to fix the non determinism.
> 
> >
> >Could also sombody please share what kind of CI pipeline is there 
> >implemented on applying the patch on the mainter's side?
> 
> The maentainer is responsible for testing the patch (at a minimum that it 
> builds), and sometimes we forget to check it's reproducible. The only 
> automated thing is to check the patch is signed.
> 
> >
> >Thanks
> >Petr
> >
> >PS: Yes, I do have more questions regading how Guix works lately. I tried to 
> >search the mailing list but haven't found answers to them
> 

I'm not always great with diffoscope, but it turns out this one isn't
too bad to read.

(ins)efraim@3900XT ~ [env]$ diffoscope 
/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2 
/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check
--- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2
+++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check
│   --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share
├── +++ 
/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share
│ │   --- 
/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo
│ ├── +++ 
/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo
│ │ │   --- 
/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo/registry
│ │ ├── +++ 
/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo/registry
│ │ │ │   --- 
/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo/registry/brotli-sys-0.3.2.crate
│ │ │ ├── +++ 
/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo/registry/brotli-sys-0.3.2.crate
│ │ │ │ ├── brotli-sys-0.3.2.crate-content
│ │ │ │ │ ├── file list
│ │ │ │ │ │ @@ -1,8 +1,8 @@
│ │ │ │ │ │ --rw-r--r--   0        0        0     1123 2021-08-05 
07:38:54.000000 brotli-sys-0.3.2/Cargo.toml
│ │ │ │ │ │ +-rw-r--r--   0        0        0     1123 2021-08-05 
07:38:41.000000 brotli-sys-0.3.2/Cargo.toml
│ │ │ │ │ │  -rw-r--r--   0      999    30000     1122 1970-01-01 
00:00:00.000000 brotli-sys-0.3.2/Cargo.toml.orig
│ │ │ │ │ │  -rw-r--r--   0      999    30000     2168 2017-06-30 
05:59:10.000000 brotli-sys-0.3.2/brotli/common/constants.h
│ │ │ │ │ │  -rw-r--r--   0      999    30000   471278 2017-09-19 
18:16:36.000000 brotli-sys-0.3.2/brotli/common/dictionary.c
│ │ │ │ │ │  -rw-r--r--   0      999    30000     1499 2017-09-19 
18:16:36.000000 brotli-sys-0.3.2/brotli/common/dictionary.h
│ │ │ │ │ │  -rwxr-xr-x   0      999    30000      595 2017-09-19 
18:16:36.000000 brotli-sys-0.3.2/brotli/common/version.h
│ │ │ │ │ │  -rw-r--r--   0      999    30000     1259 2017-06-30 
05:59:10.000000 brotli-sys-0.3.2/brotli/dec/bit_reader.c
│ │ │ │ │ │  -rw-r--r--   0      999    30000    12006 2017-06-30 
05:59:10.000000 brotli-sys-0.3.2/brotli/dec/bit_reader.h
│ │ │ │ ├── stat {}
│ │ │ │ │ @@ -1,8 +1,8 @@
│ │ │ │ │
│ │ │ │ │ -  Size: 406369       Blocks: 800        IO Block: 4096   regular file
│ │ │ │ │ +  Size: 406370       Blocks: 800        IO Block: 4096   regular file
│ │ │ │ │  Links: 1
│ │ │ │ │  Access: (0444/-r--r--r--)  Uid: (    0/    root)   Gid: (    0/    
root)
│ │ │ │ │
│ │ │ │ │  Modify: 1970-01-01 00:00:01.000000000 +0000

Here the difference is the timestamp on Cargo.toml. Cargo.toml gets
regenerated during the build process (likely when we run `cargo build`)
and then during the install phase we create a tarball of the source and
put that and an unpacked version in %out/share/cargo as a step toward
switching from #:cargo-inputs to just 'inputs'.

Ideally we'd modify the install phase or add a new phase to reset some
of the timestamps. Unfortunately, modifying the cargo-build-system would
result in the rebuild of ~2500 packages, or ~150, depending on how you
count.

I can see about queueing up something for core-updates-frozen, which
will make its way back to the master branch soonish™.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

signature.asc
Description: PGP signature