help-guix
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Non deterministic package


From: phodina
Subject: Re: Non deterministic package
Date: Mon, 09 Aug 2021 15:13:20 +0000

> I've become involved in the rust packaging effort so I'll jump in with
>
> some specifics.
>
> On Wed, Aug 04, 2021 at 03:50:00PM -0400, Julien Lepiller wrote:
>
> > Le 4 août 2021 15:14:16 GMT-04:00, phodina via help-guix@gnu.org a écrit :
> >
> > > Hi,
> > >
> > > I came across a package in the Guix repository which fails to build. The 
> > > package in question here is rust-brotli-sys-0.3. So the obvious question 
> > > comes up. How does this happen?
> > >
> > > I just ran =guix build rust-brotli-sys= with the following output:
> > >
> > > ...
> > >
> > > phase `reset-gzip-timestamps' succeeded after 0.0 seconds starting phase` 
> > > compress-documentation'
> > >
> > > phase `compress-documentation' succeeded after 0.0 seconds guix build: 
> > > error: derivation` 
> > > /gnu/store/kf8kwl7yn8jqpwc9p97bw8vs5g5bvas4-rust-brotli-sys-0.3.2.drv' 
> > > may not be deterministic: output 
> > > `/gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2' 
> > > differs
> > >
> > > Also this is my guix channel:
> > >
> > > guix:
> > >
> > > repository URL: https://git.savannah.gnu.org/git/guix.git
> > >
> > > branch: master
> > >
> > > commit: b95fb85f593d377864fc1bfb3c118e54d03783be
> > >
> > > I tried to run the build with the flag --check and multiple rounds, but 
> > > got the same result.
> > >
> > > So could it be caused by not running =./pre-inst-env guix build 
> > > --rounds=2 <PKG>= when submitting the patch?
> >
> > You need to use. /pre-inst-env to test your patches, otherwise you'll only 
> > run your current guix. For an added package, it will be missing, for an 
> > updated package it will not build the newer version, etc.
> >
> > > Or when somebody updates some pkg that this pkg depends on and does not 
> > > run =./pre-inst-env guix --refresh --list-dependent= and then check the 
> > > dependent pkgs?
> >
> > It can also happen when only the current day is recorded in the output: if 
> > you use —rounds=2, it is likely run on the same day, so the output will be 
> > the same (don't bother though, last time I found such an issue it was in 
> > january because only the year was recorded ^^")
> >
> > You should now use -K to keep the different version and run diffoscope to 
> > analyze the differences. Then we should be able to fix the non determinism.
> >
> > > Could also sombody please share what kind of CI pipeline is there 
> > > implemented on applying the patch on the mainter's side?
> >
> > The maentainer is responsible for testing the patch (at a minimum that it 
> > builds), and sometimes we forget to check it's reproducible. The only 
> > automated thing is to check the patch is signed.
> >
> > > Thanks
> > >
> > > Petr
> > >
> > > PS: Yes, I do have more questions regading how Guix works lately. I tried 
> > > to search the mailing list but haven't found answers to them
>
> I'm not always great with diffoscope, but it turns out this one isn't
>
> too bad to read.
>
> (ins)efraim@3900XT ~ [env]$ diffoscope 
> /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2 
> /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check
>
> --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2
>
> +++ /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check
>
> │ --- /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share
>
> ├── +++ 
> /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share
>
> │ │ --- 
> /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo
>
> │ ├── +++ 
> /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo
>
> │ │ │ --- 
> /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo/registry
>
> │ │ ├── +++ 
> /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo/registry
>
> │ │ │ │ --- 
> /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2/share/cargo/registry/brotli-sys-0.3.2.crate
>
> │ │ │ ├── +++ 
> /gnu/store/yrrdr6rrc1s4b9lf7wz40c2gnv5kvxff-rust-brotli-sys-0.3.2-check/share/cargo/registry/brotli-sys-0.3.2.crate
>
> │ │ │ │ ├── brotli-sys-0.3.2.crate-content
>
> │ │ │ │ │ ├── file list
>
> │ │ │ │ │ │ @@ -1,8 +1,8 @@
>
> │ │ │ │ │ │ --rw-r--r-- 0 0 0 1123 2021-08-05 07:38:54.000000 
> brotli-sys-0.3.2/Cargo.toml
>
> │ │ │ │ │ │ +-rw-r--r-- 0 0 0 1123 2021-08-05 07:38:41.000000 
> brotli-sys-0.3.2/Cargo.toml
>
> │ │ │ │ │ │ -rw-r--r-- 0 999 30000 1122 1970-01-01 00:00:00.000000 
> brotli-sys-0.3.2/Cargo.toml.orig
>
> │ │ │ │ │ │ -rw-r--r-- 0 999 30000 2168 2017-06-30 05:59:10.000000 
> brotli-sys-0.3.2/brotli/common/constants.h
>
> │ │ │ │ │ │ -rw-r--r-- 0 999 30000 471278 2017-09-19 18:16:36.000000 
> brotli-sys-0.3.2/brotli/common/dictionary.c
>
> │ │ │ │ │ │ -rw-r--r-- 0 999 30000 1499 2017-09-19 18:16:36.000000 
> brotli-sys-0.3.2/brotli/common/dictionary.h
>
> │ │ │ │ │ │ -rwxr-xr-x 0 999 30000 595 2017-09-19 18:16:36.000000 
> brotli-sys-0.3.2/brotli/common/version.h
>
> │ │ │ │ │ │ -rw-r--r-- 0 999 30000 1259 2017-06-30 05:59:10.000000 
> brotli-sys-0.3.2/brotli/dec/bit_reader.c
>
> │ │ │ │ │ │ -rw-r--r-- 0 999 30000 12006 2017-06-30 05:59:10.000000 
> brotli-sys-0.3.2/brotli/dec/bit_reader.h
>
> │ │ │ │ ├── stat {}
>
> │ │ │ │ │ @@ -1,8 +1,8 @@
>
> │ │ │ │ │
>
> │ │ │ │ │ - Size: 406369 Blocks: 800 IO Block: 4096 regular file
>
> │ │ │ │ │ + Size: 406370 Blocks: 800 IO Block: 4096 regular file
>
> │ │ │ │ │ Links: 1
>
> │ │ │ │ │ Access: (0444/-r--r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
>
> │ │ │ │ │
>
> │ │ │ │ │ Modify: 1970-01-01 00:00:01.000000000 +0000
>
> Here the difference is the timestamp on Cargo.toml. Cargo.toml gets
>
> regenerated during the build process (likely when we run `cargo build`)
>
> and then during the install phase we create a tarball of the source and
>
> put that and an unpacked version in %out/share/cargo as a step toward
>
> switching from #:cargo-inputs to just 'inputs'.
>
> Ideally we'd modify the install phase or add a new phase to reset some
>
> of the timestamps. Unfortunately, modifying the cargo-build-system would
>
> result in the rebuild of ~2500 packages, or ~150, depending on how you
>
> count.
>
> I can see about queueing up something for core-updates-frozen, which
>
> will make its way back to the master branch soonish™.
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Efraim Flashner efraim@flashner.co.il אפרים פלשנר
>
> GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
>
> Confidentiality cannot be guaranteed on emails sent or received unencrypted

Thanks Julien and Efraim. I didn't know about diffoscope, I see it's described 
in the manual[1].

I agree that the package build might build if the second build runs the same 
day. I'll add this tool to my list of checks for submitting new patches.

I use ./pre-inst-env to build and lint the package in question. Not the guix 
binary in my system itself. So I was little bit surprised that this package 
didn't build as it was already part of the Guix monorepo.Also this was the 
first time I encountered such an issue.

[1]: https://guix.gnu.org/manual/en/html_node/Invoking-guix-challenge.html




reply via email to

[Prev in Thread] Current Thread [Next in Thread]