gnu-misc-discuss
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Risks of deterministic builds


From: Jan Nieuwenhuizen
Subject: Re: Risks of deterministic builds
Date: Wed, 14 Apr 2021 10:26:44 +0200
User-agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)

Jean Louis writes:

> * Jan Nieuwenhuizen <janneke@gnu.org> [2021-04-08 16:43]:
>> Martin writes:
>> 
>> > Maybe freedom in "free software" shouldn't require from the code to be
>> > open neither. Let's just blindly trust some saint developers who
>> > cannot even control their own binaries. Actually today we are closer
>> > and closer to that sad scenario like never before in the history,
>> > because in fact most of the open-source and GNU "free software"
>> > nowadays base on blackboxed binary seeds that cannot be verified by
>> > the users not even by the core developers.
>> 
>> The bootstrappable project, GNU Mes and GNU Guix are working to fix that
>> 
>>     
>> https://guix.gnu.org/en/blog/2020/guix-further-reduces-bootstrap-seed-to-25/
>>     https://fosdem.org/2021/schedule/event/gnumes/
>
> Jan, in regards to this, how to start with first stage, how are you
> entering it?
> Will you use for example Emacs and hexl-mode to enter bytes?

I am not, I'm downloading the initial 357-byte binary from
'bootstrap-seeds'

    
https://raw.githubusercontent.com/oriansj/bootstrap-seeds/master/POSIX/x86/hex0-seed

and then run this script

    
https://github.com/oriansj/stage0-posix/blob/master/x86/mescc-tools-seed-kaem.kaem

See below for an overview and the next steps.

Note that we are only in the 3rd iteration of removing binary seeds from
the bootstrap; so in a way this project is just starting.  A year ago,
Ludovic Courtès already prototyped a step to build the first packages in
the initial ramdisk, such as to remove dependencies on the Guix build
daemon.

Also note that major forces are going into "unbootstrappable" projects,
that may even undo the small steps that we have just achieved.  It has
become much harder to bootstrap GCC from source then 10 years ago: we
need unmaintained earlier versions of GCC to do so.  Dependencies on
Python, Rust are moving up the package graph.

I choose not to believe that "nobody cares" about bootstrappability,
i.e., binary-blob-free computing, I just think we need to raise
awareness about it if we don't want to lose it.

Greetings,
Janneke


The bootstrappable effort to bootstrap a full GNU/Linux system from
source consists mainly of three projects: Stage0 by Jeremiah Orians
and others, GNU Mes by Janneke and others, and the actual bootstrap
packaging and deployment in GNU Guix:

    https://guix.gnu.org/en/manual/en/html_node/Bootstrapping.html#Bootstrapping

Efforts are ongoing to implement such a bootstrap in other
distributions.

The reduced binary seed bootstrap was developed in GNU Guix in and is
used in production

    https://guix.gnu.org/blog/2019/guix-reduces-bootstrap-seed-by-50/
    https://guix.gnu.org/blog/2020/guix-further-reduces-bootstrap-seed-to-25

There have been several efforts to avoid using a package manager such
as GNU Guix and most hove been abandoned.  Possibly because it's fun
to start such a thing, and pretty easy even to do the first steps
without isolated builds, (cross-)build, or packaging support.

The first working development version of 'wip-full-source-bootstrap'
in Guix triggered a yet another effort like that, the live-bootstrap

    https://github.com/fosslinux/live-bootstrap

The full source bootstrap is not in production; a development preview
lives here

    
https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/commencement.scm?h=wip-full-source-bootstrap

The package graph for this bootstrap; the bottom of the guix package
graph as generated by 'guix graph' looks like this

    
https://git.savannah.gnu.org/cgit/mes.git/plain/doc/talks/fosdem21/gcc-core-mesboot0-fsb-no-guile-unversioned.png?h=wip-talks

which can be split into several stages

    
https://git.savannah.gnu.org/cgit/mes.git/plain/doc/talks/fosdem21/stage-0.png?h=wip-talks
    
https://git.savannah.gnu.org/cgit/mes.git/plain/doc/talks/fosdem21/stage-1.png?h=wip-talks
    
https://git.savannah.gnu.org/cgit/mes.git/plain/doc/talks/fosdem21/stage-2.png?h=wip-talks
    
https://git.savannah.gnu.org/cgit/mes.git/plain/doc/talks/fosdem21/stage-mes.png?h=wip-talks

it may be easier to read that from the slides of my FOSDEM'21 talk

    
https://fosdem.org/2021/schedule/event/gnumes/attachments/slides/4531/export/events/attachments/gnumes/slides/4531/gnu_mes_fosdem21.pdf

You can see it all starts with 'bootstrap-seeds'.  You can "build" that
package (the seeds are just a download, of course) by (checking out
the guix wip-full-source-bootstrap branch, building it) and, doing

    $ ./pre-inst-env guix build -e '(@@ (gnu packages commencement) 
bootstrap-seeds)'
    /gnu/store/sg0fxjd0pyfn5sm0m33227i3rz32l6z6-bootstrap-seeds-1.0.0

--8<---------------cut here---------------start------------->8---
  (define m2-planet-boot
    ;; The initial bootstrap package: no binary inputs except for a 357 byte
    ;; binary seed: `x86/hex0-seed'.
[..]
    (let* ((bootstrap-seeds (assoc-ref %build-inputs "bootstrap-seeds"))
           (mescc-tools-seed (assoc-ref %build-inputs "mescc-tools-seed"))
           (mescc-tools (assoc-ref %build-inputs "mescc-tools"))
           (source (assoc-ref %build-inputs "source"))
           (tar (assoc-ref %build-inputs "bootar"))
           (out (assoc-ref %outputs "out"))
           (bindir (string-append out "/bin")))
      (invoke "tar" "xvf" mescc-tools-seed)
      (chdir
       (string-append "mescc-tools-seed-" ,mescc-tools-seed-version))
      (copy-recursively bootstrap-seeds "bootstrap-seeds")
      (invoke "tar" "xvf" mescc-tools)
      (rmdir "mescc-tools")
      (symlink (string-append "mescc-tools-" ,mescc-tools-version)
               "mescc-tools")
      (invoke "tar" "xvf" source)
      (rmdir "M2-Planet")
      (symlink (string-append "m2-planet-" ,version) "M2-Planet")
      (mkdir-p bindir)
      ;; XXX TODO: Update mescc-tools-seed
      (copy-file "M2-Planet/test/common_x86/x86_defs.M1" "x86/x86_defs.M1")
      (with-directory-excursion "x86"
        (invoke "../bootstrap-seeds/POSIX/x86/kaem-optional-seed"
                "mescc-tools-seed-kaem.kaem")
        (invoke "../bootstrap-seeds/POSIX/x86/kaem-optional-seed"
                "mescc-tools-mini-kaem.kaem")
        (invoke "../bootstrap-seeds/POSIX/x86/kaem-optional-seed"
                "mescc-tools-full-kaem.kaem"))
      (with-directory-excursion "bin"
        (install-file "hex2" bindir)
        (install-file "M1" bindir)
        (install-file "blood-elf" bindir)
        (install-file "kaem" bindir)
        (install-file "get_machine" bindir)
        (install-file "M2-Planet" bindir))
      #t)))))))
--8<---------------cut here---------------end--------------->8---

and you'll see the 357-byte hex0-seed in there.  This is used to build
the 'm2-planet-boot' package which uses kaem scripts (any shell will
do) to build hex0, hex1, M0, M1, hex2 and m2-planet.

As you can see, the very first step is running this script

    
https://raw.githubusercontent.com/oriansj/mescc-tools-seed/master/x86/mescc-tools-seed-kaem.kaem

which uses this 357-byte seed

    
https://raw.githubusercontent.com/oriansj/bootstrap-seeds/master/POSIX/x86/hex0-seed

built using hex0 itself from

    
https://raw.githubusercontent.com/oriansj/bootstrap-seeds/master/POSIX/x86/hex0_x86.hex0

hex0 can be approximated with:

    sed 's/[;#].*$//g' $input_file | xxd -r -p > $output_file

kaem-optional-seed can be replaced by any shell of your choice as it
is just a minimal shell written in hex0.

Then onto

    
https://raw.githubusercontent.com/oriansj/mescc-tools-seed/master/x86/mescc-tools-mini-kaem.kaem

and

    
https://raw.githubusercontent.com/oriansj/mescc-tools-seed/master/x86/mescc-tools-full-kaem.kaem

and so on; or simply:

    $ ./pre-inst-env guix build -e '(@@ (gnu packages commencement) 
m2-planet-boot)'
    /gnu/store/qxsd6zg910zjgwsm13c39hys2g931ij8-m2-planet-boot-1.7.0-31-g358b6cf

then onto mes-boot

    $ ./pre-inst-env guix build -e '(@@ (gnu packages commencement) mes-boot)'
    /gnu/store/dfwkand25k6nsmrk85p8hsa1yq74y314-mes-boot-0.22-305-g2ab4c5c67

and so on.  In fact, each time you build or install a package using
Guix, these (bootstrap) dependencies are resolved and -if necessary- all
built automagically.

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com



reply via email to

[Prev in Thread] Current Thread [Next in Thread]