bug#57593: Racket 8.6 is not reproducible

[Philip McGrath]

Hi,

On Mon, Sep 5, 2022, at 6:04 AM, Ludovic Courtès wrote:
> Hey,
>
> Just noticed that Racket does not build in a reproducible fashion:
>
> --8<---------------cut here---------------start------------->8---
> $ guix describe
> Generation 227  Sep 04 2022 23:39:52    (current)
>   guix aae98c2
>     repository URL: https://git.savannah.gnu.org/git/guix.git
>     branch: master
>     commit: aae98c297214f87eb45302863adb021078c41a6f
> $ guix challenge racket | head -40
> /gnu/store/av255rh362283i1zaiq9rz4rpli69j59-racket-8.6 contents differ:
>   local hash: 1w4dnkvpbbrgfasyq8x1cbqw58jzqsny17ms5l1fb1h6iid38bs1
>   
> https://ci.guix.gnu.org/nar/lzip/av255rh362283i1zaiq9rz4rpli69j59-racket-8.6: 
> 1lnxklizpnc599w7n2svb1jaw595wranm9aagd2928fcbiaavbr6
>   differing files:
>     /lib/racket/pkgs/2d-doc/scribblings/compiled/2d_scrbl.dep
>     /lib/racket/pkgs/2d-doc/scribblings/compiled/2d_scrbl.zo

Unfortunately, this is a known issue upstream.

One part of the problem is the way that Chez Scheme uses "type 4" (random) 
UUIDs to implement 'gensym'. There's some upstream discussion at 
<https://github.com/cisco/ChezScheme/issues/585>, including an idea from me 
about "type 5" UUIDs as part of a possible solution.

That may well be the only issue with the core compiler for Racket CS. I know 
effort has been taken to make Racket's front-end deterministic, and Kent Dybvig 
says that:

> Chez Scheme compiles are reproducible modulo the globally unique
> names of generated symbols (gensyms), assuming macros don't do
> things like introduce time stamps or random expressions into their output.
> Because of differences in gensym names, the files produced by multiple
> runs are not byte-by-byte identical but should compare equal via
> the system primitive $fasl-file-equal? See, for example, the use
> of $fasl-file-equal? in the s/Mf-base checkboot target.

In the output you sent, the ".zo" files contain Chez machine code with a 
Racket-specific wrapper, and the ".dep" files contain hashes of dependencies 
(so they differ when the ".zo" files they track differ).

Debian reports at 
<https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/racket.html>,
 discussing Racket 8.2+dfsg1-2:

> Build path captured in html documentation. docindex.sqlite
> contains content in arbitrary order. strings in binaries in
> arbitrary order.

Those issues don't appear to affect Guix: some might be caused by Debian 
conflating "build" and "install" for Racket packages, and, in any case, they 
seem related to the "scribble" and/or "racket-index" packages, not the core 
compiler.

IIRC, there were some reproducibility issues with the Racket BC bytecode 
compiler. If we can build reproducibly with CS but not BC, it might work out to 
disable compilation for the bootstrapping BC compilers, or maybe to configure 
them to use the machine-independent format (basically, a serialized form of 
what the front-end would hand off to the back-end), at the cost of some 
performance.

I hope to work on these issues, but I plan to work on a 'racket-build-system' 
first. Among other things, I hope it will be easier to investigate issues when 
Racket packages can be built independently.

-Philip