bug#36747: Official MesCC bootstrap binaries differ from my locally built ones

[Marius Bakke]

Mark H Weaver <address@hidden> writes:

> Hi Marius,
>
> Marius Bakke <address@hidden> writes:
>
>> Marius Bakke <address@hidden> writes:
>>
>>> Jan Nieuwenhuizen <address@hidden> writes:
>>>
>>>> Mark H Weaver writes:
>>>>
>>>> Hi Mark,
>>>>
>>>>>> I called that `wip-binaries', @master from three weeks ago.
>>>>>
>>>>> Thank you, that was a good start.  I found that some additional patches
>>>>> were needed to match the bootstrap binaries that 'core-updates' is
>>>>> currently based on.
>>>>>
>>>>> I ended up deleting and repushing a revised 'wip-binaries' to Savannah.
>>>>> It includes slightly modified versions of the two commits you had
>>>>> included, as well as some additional cherry-picked commits of yours to
>>>>> update mescc-tools and add linux-libre-headers-bootstrap-tarball, and a
>>>>> few of my own.
>>>>
>>>> Very nice.
>>>>
>>>>> I built the new bootstrap tarballs at the new 'wip-binaries', commit
>>>>> c67becb31c30a5cd7685f166970ac4793e3a34a9, and here's what I got:
>>>>>
>>>>> mhw@jojen ~/guix-wip-binaries$ git describe
>>>>> v1.0.1-2404-gc67becb31c
>>>>> mhw@jojen ~/guix-wip-binaries$ ./pre-inst-env guix build 
>>>>> --system=i686-linux bootstrap-tarballs
>>>>> /gnu/store/bg086i2qw1fn2jgbd15d9v91hyjrjsb2-bootstrap-tarballs-0
>>>>> mhw@jojen ~/guix-wip-binaries$ cd 
>>>>> /gnu/store/bg086i2qw1fn2jgbd15d9v91hyjrjsb2-bootstrap-tarballs-0
>>>>> mhw@jojen 
>>>>> /gnu/store/bg086i2qw1fn2jgbd15d9v91hyjrjsb2-bootstrap-tarballs-0$ 
>>>>> sha256sum *
>>>>> 3e50c070a100b6bcf84c4bf5c868f9cd0a9fd1570f5d82fbfb78f8411959091b  
>>>>> guile-static-stripped-2.2.4-i686-linux.tar.xz
>>>>> 1acd8f83e27d2fac311a5ca78e9bf11a9a1638b82469870d5c854c4e7afaa26a  
>>>>> linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
>>>>> 021543d9bb6af55f39e68d69692e3cb74646ced2cad0bb9ac0047ef81e9d7330  
>>>>> mescc-tools-static-stripped-0.5.2-0.bb062b0-i686-linux.tar.xz
>>>>> fb32090071b39fc804fb9a7fba96f0bc5eb844a0efd268fb24c42e6bfa959de0  
>>>>> mes-minimal-stripped-0.19-i686-linux.tar.xz
>>>>> c80cdd17b0a24eebdd75570ff72c4ec06e129bd702ac008186b57f6301c448e7  
>>>>> static-binaries-0-i686-linux.tar.xz
>>>>
>>>>> Can you try "guix build --system=i686-linux bootstrap-tarballs" at the
>>>>> new 'wip-binaries' branch and see if you get the same results?
>>>>
>>>> Yes, on c67becb31c30a5cd7685f166970ac4793e3a34a9 running
>>>> "./pre-inst-env guix build --system=i686-linux bootstrap-tarballs" gives 
>>>> me exactly this,
>>>> also for guile-static-stripped! \o/
>>>>
>>>>> Also, I have a question: One of the changes I made to 'wip-binaries' was
>>>>> to update mescc-tools to 0.5.2-0.bb062b0, to match the
>>>>> %bootstrap-mescc-tools that's currently being used in 'core-updates'.
>>>>>
>>>>> However, I noticed that you have also apparently built the official
>>>>> release of mescc-tools-0.5.2, which is on your site:
>>>>>
>>>>>   
>>>>> http://lilypond.org/janneke/guix/20190722/mescc-tools-static-stripped-0.5.2-i686-linux.tar.xz
>>>>>
>>>>> and that this tarball is identical to the build output of the later git
>>>>> commit: mescc-tools-static-stripped-0.5.2-0.bb062b0-i686-linux.tar.xz.
>>>>>
>>>>> With this in mind, could we just use 0.5.2?  What changed between 0.5.2
>>>>> and 0.5.2-0.bb062b0, and what was the rationale for updating to bb062b0?
>>>>
>>>> Good catch.  We probably can, we might try that.
>>>>
>>>> I think the need for updating to bb062b0 has been removed during the
>>>> review of the integration of the reduced binary seed bootstrap into
>>>> core-updates by Ludovic.
>>>>
>>>> For historical reasons, I think this mescc-tools commit
>>>>
>>>> --8<---------------cut here---------------start------------->8---
>>>> commit c184e95096881a13f29ebd7fc507fe305d3d8de5 (gitlab/janneke, janneke)
>>>> Author: Jan Nieuwenhuizen <address@hidden>
>>>> Date:   Thu Oct 4 22:03:31 2018 +0200
>>>>
>>>>     build.sh: Update for mes 0.18.
>>>> --8<---------------cut here---------------end--------------->8---
>>>>
>>>> was needed at a time that we did not have mescc-tools or mes in
>>>> bootstrap tarballs.  We built bootstrap variants of mescc-tools and mes
>>>> using a externally (outside fo Guix) built mescc-tools-seed and
>>>> (an almost pure ASCII) mes-seed.
>>>
>>> I tried building the i686 bootstrap tarballs from wip-binaries with this
>>> additional patch:
>>>
>>> diff --git a/gnu/packages/mes.scm b/gnu/packages/mes.scm
>>> index e298cb05c1..380cac6c88 100644
>>> --- a/gnu/packages/mes.scm
>>> +++ b/gnu/packages/mes.scm
>>> @@ -139,33 +139,31 @@ Guile.")
>>>      (license gpl3+)))
>>>  
>>>  (define-public mescc-tools
>>> -  (let ((commit "bb062b0da7bf2724ca40f9002b121579898d4ef7")
>>> -        (revision "0")
>>> -        (version "0.5.2"))
>>> -    (package
>>> -      (name "mescc-tools")
>>> -      (version (string-append version "-" revision "." (string-take commit 
>>> 7)))
>>> -      (source (origin
>>> -                (method url-fetch)
>>> -                (uri (string-append
>>> -                      
>>> "https://git.savannah.nongnu.org/cgit/mescc-tools.git/snapshot/";
>>> -                      name "-" commit
>>> -                      ".tar.gz"))
>>> -                (sha256
>>> -                 (base32
>>> -                  
>>> "1h6j57wyf91i42b26f8msbv6451cw3nm4nmpl1fckp9c7vi8mwkh"))))
>>> -      (build-system gnu-build-system)
>>> -      (supported-systems '("i686-linux" "x86_64-linux"))
>>> -      (arguments
>>> -       `(#:make-flags (list (string-append "PREFIX=" (assoc-ref %outputs 
>>> "out")))
>>> -         #:test-target "test"
>>> -         #:phases (modify-phases %standard-phases
>>> -                    (delete 'configure))))
>>> -      (synopsis "Tools for the full source bootstrapping process")
>>> -      (description
>>> -       "Mescc-tools is a collection of tools for use in a full source
>>> +  (package
>>> +    (name "mescc-tools")
>>> +    (version "0.5.2")
>>> +    (source (origin
>>> +              (method url-fetch)
>>> +              (uri (string-append
>>> +                    
>>> "https://git.savannah.nongnu.org/cgit/mescc-tools.git/snapshot/";
>>> +                    name "-Release_" version
>>> +                    ".tar.gz"))
>>> +              (file-name (string-append name "-" version ".tar.gz"))
>>> +              (sha256
>>> +               (base32
>>> +                "01x7bhmgwyf6mc2g1hcvibhps98nllacqm4f0j5l51b1mbi18pc2"))))
>>> +    (build-system gnu-build-system)
>>> +    (supported-systems '("i686-linux" "x86_64-linux"))
>>> +    (arguments
>>> +     `(#:make-flags (list (string-append "PREFIX=" (assoc-ref %outputs 
>>> "out")))
>>> +       #:test-target "test"
>>> +       #:phases (modify-phases %standard-phases
>>> +                  (delete 'configure))))
>>> +    (synopsis "Tools for the full source bootstrapping process")
>>> +    (description
>>> +     "Mescc-tools is a collection of tools for use in a full source
>>>  bootstrapping process.  It consists of the M1 macro assembler, the hex2
>>>  linker, the blood-elf symbol table generator, the kaem shell, exec_enable 
>>> and
>>>  get_machine.")
>>>      (home-page "https://savannah.nongnu.org/projects/mescc-tools";)
>>> -    (license gpl3+))))
>>> +    (license gpl3+)))
>
> I guess this is equivalent to reverting commit
> 78ced7975b0665e810834391d826c9f0ef7277e1 on the 'wip-binaries' branch,
> yes?

Indeed.

>>> And got this result:
>>>
>>> $ cd /gnu/store/rdwyr8mh7dvhfkb5g4cws6q40hp23rbi-bootstrap-tarballs-0
>>> $ sha256sum *
>>> 3e50c070a100b6bcf84c4bf5c868f9cd0a9fd1570f5d82fbfb78f8411959091b  
>>> guile-static-stripped-2.2.4-i686-linux.tar.xz
>>> 1acd8f83e27d2fac311a5ca78e9bf11a9a1638b82469870d5c854c4e7afaa26a  
>>> linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
>>> 021543d9bb6af55f39e68d69692e3cb74646ced2cad0bb9ac0047ef81e9d7330  
>>> mescc-tools-static-stripped-0.5.2-i686-linux.tar.xz
>>> fb32090071b39fc804fb9a7fba96f0bc5eb844a0efd268fb24c42e6bfa959de0  
>>> mes-minimal-stripped-0.19-i686-linux.tar.xz
>>> c80cdd17b0a24eebdd75570ff72c4ec06e129bd702ac008186b57f6301c448e7  
>>> static-binaries-0-i686-linux.tar.xz
>
> Great, looks good to me!
>
>>> I also merged the branch to core-updates and reverted the bash patch,
>
> I think that 'bash-4.4-linux-pgrp-pipe.patch', adapted to 5.0, should be
> applied to all 'bash' packages in 'core-updates'.  To be more precise,
> the patch should be applied to the main 'bash' package in bash.scm,
> inherited from all other bash packages, so that PGRP_PIPE is set
> unconditionally set on systems based on Linux (the kernel), regardless
> of the kernel version running on the build machine.

Got it.

>>> which produced this derivation for "guix build -d -s i686-linux
>>> bootstrap-tarballs":
>>>
>>> /gnu/store/ld82vli1msfrlimjaryznrqcwm0jc5ii-bootstrap-tarballs-0.drv
>>
>> Here are the hashes from this derivation:
>>
>> $ cd /gnu/store/srsqilb3g70r8c7vma0gpam21z1zmg2w-bootstrap-tarballs-0
>> $ sha256sum *
>> 4e5b219be4d9ad4d125f17b8a8a991e78be3908aadc8d22d1a115e96ec856279  
>> guile-static-stripped-2.2.6-i686-linux.tar.xz
>> 9e34b26526b184912b0d97df96e316151fd97ede24aec2f22dd13ed33438b2e4  
>> linux-libre-headers-stripped-4.19.56-i686-linux.tar.xz
>> 7db07a7097a7920e17f6f1794098b9f29bc3522c36a33cf797123f64581cdc34  
>> mescc-tools-static-0.5.2-i686-linux.tar.xz
>> 3a9b050c1a2b61bb1cbed16ee07aecdc83e9097ce4a914623aa1b9808abe57f4  
>> mes-minimal-stripped-0.19-i686-linux.tar.xz
>> 1ccb4f39656eb977706db7a4aa811461785200b0a11e847035bb89fef91b85ab  
>> static-binaries-0-i686-linux.tar.xz
>
> What was the purpose of building these?  'core-updates' is built upon on
> the earlier, unverified, reduced binary seed bootstrap binaries.

I wanted to check that the bootstrap-tarballs machinery still worked
after merging the branch, since it was non-trivial.  Mainly to make the
commit that created them reachable forever, but maybe we don't need it.

>> @Mark: do you think we are ready to merge this now?  Can you do it?
>
> I think what needs to be done is the following:
>
> (1) commit 78ced7975b0665e810834391d826c9f0ef7277e1 on 'wip-binaries'
>     should be reverted, to downgrade mescc-tools to the 0.5.2 release.
>
> (2) The 'wip-binaries' tarballs should be uploaded to a new subdirectory
>     of <https://alpha.gnu.org/gnu/guix/bootstrap/i686-linux/>, along
>     with digital signatures, of course.  I'm talking about these in
>     particular:
>
>     3e50c070a100b6bcf84c4bf5c868f9cd0a9fd1570f5d82fbfb78f8411959091b  
> guile-static-stripped-2.2.4-i686-linux.tar.xz
>     1acd8f83e27d2fac311a5ca78e9bf11a9a1638b82469870d5c854c4e7afaa26a  
> linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
>     021543d9bb6af55f39e68d69692e3cb74646ced2cad0bb9ac0047ef81e9d7330  
> mescc-tools-static-stripped-0.5.2-i686-linux.tar.xz
>     fb32090071b39fc804fb9a7fba96f0bc5eb844a0efd268fb24c42e6bfa959de0  
> mes-minimal-stripped-0.19-i686-linux.tar.xz
>     c80cdd17b0a24eebdd75570ff72c4ec06e129bd702ac008186b57f6301c448e7  
> static-binaries-0-i686-linux.tar.xz
>
> (3) The following bootstrap packages in 'core-updates' bootstrap.scm
>     should be updated to use the new binaries above:
>
>      (a) %bootstrap-linux-libre-headers
>      (b) %bootstrap-mescc-tools
>      (c) %bootstrap-mes
>
> (4) Berlin should start rebuilding 'core-updates'.
>
> If desired, steps (3) and (4) could come before (2) if someone
> temporarily uploads the new binaries somewhere else, and adjusts
> '%bootstrap-base-urls' accordingly.  The key is for the hashes and file
> names to match what we've agreed on here, as I listed in (2) above.
>
> What do you think?

Thank you for the excellent summary.  I can look into adjusting the bash
fix for 5.0, and updating the bootstrap binary URLs and hashes.  I will
do this in a 'core-updates-next' branch.  I would also like to merge
wip-binaries into it as a final step, unless someone has objections.

Ludovic should be back in a couple of days and can hopefully take care
of the uploads.

Ricardo: can you instruct Cuirass to add a reduced jobset for
'core-updates-next', that only builds builds the "core" package subset?

signature.asc
Description: PGP signature