[bug#43160] Validate the result of our linux-libre sources clean up

[Maxim Cournoyer]

Hi Mark!

Mark H Weaver <mhw@netris.org> writes:

> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:
>> I'd like to point you to the following patches, as they touch the
>> generation of the linux-libre sources, in case they hadn't caught your
>> attention: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43160.
>
> Thanks very much for bringing this to my attention.  I do not subscribe
> to the guix-patches list, so I would not have seen this otherwise.
>
> I'm in favor of the following patches:
>
>   gnu: linux-libre: Use Python 3 in make-linux-libre-source.
>   gnu: make-linux-libre-source: Set output port buffering to line mode.
>   gnu: linux-libre: Validate that the cleaned up tarball is free of blobs.
>
> Thanks for these.  Please push them whenever you feel is appropriate.

Thanks for taking a look!  I've now done so.

> On other other hand, I'm strongly opposed to the following patch:
>
>   gnu: linux-libre: Compare generated sources against Linux-libre releases.
>
> I'm opposed to it because it would make it prohibitively difficult to
> push micro kernel updates (most of which contain potential security
> fixes) before Linux-libre has published their tarball release.

Following recent discussions, I had understood that you agreed to wait
for the Linux-libre releases before bumping our own releases.  It seems
the Linux-libre releases occur fast enough to not pose much of a
security issue; below is what I did to arrive to this conclusion.

For Linux stable, the author dates of the last releases of the version 5
series, omitting release candidates:

--8<---------------cut here---------------start------------->8---
$ git tag | grep -E '5\.[0-9]+\.' | grep -v -- '-rc' \
    | sort -t '.' -k1,1n -k2,2n -k3,3n | tail -n20 \
    | xargs -i{} git log --date='format:%c' --pretty='%ad %d' {} -1
Wed 24 Jun 2020 05:49:26 PM GMT  (tag: v5.7.6)
Tue 30 Jun 2020 04:21:22 PM GMT  (tag: v5.7.7)
Thu 09 Jul 2020 09:39:40 AM GMT  (tag: v5.7.8)
Thu 16 Jul 2020 08:13:36 AM GMT  (tag: v5.7.9)
Wed 22 Jul 2020 09:34:29 AM GMT  (tag: v5.7.10)
Wed 29 Jul 2020 10:20:01 AM GMT  (tag: v5.7.11)
Fri 31 Jul 2020 06:47:17 PM GMT  (tag: v5.7.12)
Wed 05 Aug 2020 09:58:51 AM GMT  (tag: v5.7.13)
Fri 07 Aug 2020 09:33:11 AM GMT  (tag: v5.7.14)
Tue 11 Aug 2020 03:35:42 PM GMT  (tag: v5.7.15)
Wed 19 Aug 2020 08:24:20 AM GMT  (tag: v5.7.16)
Fri 21 Aug 2020 01:07:46 PM GMT  (tag: v5.7.17)
Wed 26 Aug 2020 11:42:25 AM GMT  (tag: v5.7.18)
Thu 27 Aug 2020 09:30:50 AM GMT  (tag: v5.7.19, origin/linux-5.7.y)
Tue 11 Aug 2020 03:48:12 PM GMT  (tag: v5.8.1)
Wed 19 Aug 2020 08:27:10 AM GMT  (tag: v5.8.2)
Fri 21 Aug 2020 01:15:22 PM GMT  (tag: v5.8.3)
Wed 26 Aug 2020 11:49:20 AM GMT  (tag: v5.8.4)
Thu 27 Aug 2020 09:31:49 AM GMT  (tag: v5.8.5)
Thu 03 Sep 2020 11:29:52 AM GMT  (tag: v5.8.6, origin/linux-5.8.y)
--8<---------------cut here---------------end--------------->8---

Similarly, for Linux-libre:

--8<---------------cut here---------------start------------->8---
git tag | grep -E 'sources/v5\.[0-9]+\.' | grep -v -- '-rc' \
    | sort -t '.' -k1,1n -k2,2n -k3,3n | tail -n20 \
    | xargs -i{} git log --date='format:%c' --pretty='%ad %d' {} -1
Wed 24 Jun 2020 02:51:34 PM GMT  (tag: sources/v5.7.6-gnu)
Wed 01 Jul 2020 02:01:47 PM GMT  (tag: sources/v5.7.7-gnu)
Thu 09 Jul 2020 08:59:49 AM GMT  (tag: sources/v5.7.8-gnu)
Thu 16 Jul 2020 11:51:43 AM GMT  (tag: sources/v5.7.9-gnu)
Wed 22 Jul 2020 06:40:22 AM GMT  (tag: sources/v5.7.10-gnu)
Wed 29 Jul 2020 06:33:25 AM GMT  (tag: sources/v5.7.11-gnu)
Fri 31 Jul 2020 02:22:04 PM GMT  (tag: sources/v5.7.12-gnu)
Wed 05 Aug 2020 05:44:37 AM GMT  (tag: sources/v5.7.13-gnu)
Fri 07 Aug 2020 04:46:28 AM GMT  (tag: sources/v5.7.14-gnu)
Tue 11 Aug 2020 02:48:28 PM GMT  (tag: sources/v5.7.15-gnu)
Wed 19 Aug 2020 02:14:46 PM GMT  (tag: sources/v5.7.16-gnu)
Fri 21 Aug 2020 09:37:45 AM GMT  (tag: sources/v5.7.17-gnu)
Wed 26 Aug 2020 07:27:54 AM GMT  (tag: sources/v5.7.18-gnu)
Thu 27 Aug 2020 01:14:21 PM GMT  (tag: sources/v5.7.19-gnu)
Tue 11 Aug 2020 02:47:58 PM GMT  (tag: sources/v5.8.1-gnu)
Wed 19 Aug 2020 02:15:42 PM GMT  (tag: sources/v5.8.2-gnu)
Fri 21 Aug 2020 09:37:45 AM GMT  (tag: sources/v5.8.3-gnu)
Wed 26 Aug 2020 07:27:54 AM GMT  (tag: sources/v5.8.4-gnu)
Thu 27 Aug 2020 01:14:21 PM GMT  (tag: sources/v5.8.5-gnu)
Thu 03 Sep 2020 07:14:30 AM GMT  (tag: sources/v5.8.6-gnu)
--8<---------------cut here---------------end--------------->8---

While the author dates of the commits don't appear to be very precise
(some Linux-libre commits would have occurred before their Linux
counterpart), we can at least see that each Linux release was met with a
Linux-libre on the same day for all except the 5.7.7 release.

Also, if we compare with our own Linux-libre update timings:

--8<---------------cut here---------------start------------->8---
git log --grep 'gnu: linux-libre: Update to 5' --date='format:%c' \
    --pretty='%ad %s' | head -n20 | sort -r -t '.' -k1,1n -k2,2n -k3,3n
Thu 11 Jun 2020 04:15:35 PM GMT gnu: linux-libre: Update to 5.4.46.
Thu 18 Jun 2020 12:39:23 AM GMT gnu: linux-libre: Update to 5.4.47
Mon 22 Jun 2020 09:02:33 PM GMT gnu: linux-libre: Update to 5.4.48.
Wed 24 Jun 2020 09:08:00 PM GMT gnu: linux-libre: Update to 5.4.49.
Wed 01 Jul 2020 01:31:06 PM GMT gnu: linux-libre: Update to 5.4.50.
Thu 09 Jul 2020 04:40:27 PM GMT gnu: linux-libre: Update to 5.4.51.
Thu 16 Jul 2020 03:37:05 PM GMT gnu: linux-libre: Update to 5.4.52.
Thu 23 Jul 2020 12:28:46 AM GMT gnu: linux-libre: Update to 5.4.53.
Wed 29 Jul 2020 05:14:00 PM GMT gnu: linux-libre: Update to 5.4.54.
Sat 01 Aug 2020 12:07:08 AM GMT gnu: linux-libre: Update to 5.4.55.
Wed 05 Aug 2020 03:21:53 PM GMT gnu: linux-libre: Update to 5.4.56.
Sat 01 Aug 2020 12:39:30 PM GMT gnu: linux-libre: Update to 5.7.12.
Fri 07 Aug 2020 09:37:11 PM GMT gnu: linux-libre: Update to 5.7.14.
Tue 11 Aug 2020 05:34:48 PM GMT gnu: linux-libre: Update to 5.7.15.
Wed 19 Aug 2020 07:35:03 PM GMT gnu: linux-libre: Update to 5.7.16.
Thu 20 Aug 2020 04:03:46 PM GMT gnu: linux-libre: Update to 5.8.2.
Fri 21 Aug 2020 09:01:17 PM GMT gnu: linux-libre: Update to 5.8.3.
Wed 26 Aug 2020 04:01:11 PM GMT gnu: linux-libre: Update to 5.8.4.
Thu 27 Aug 2020 04:13:32 PM GMT gnu: linux-libre: Update to 5.8.5.
Thu 03 Sep 2020 01:56:31 PM GMT gnu: linux-libre: Update to 5.8.6.
--8<---------------cut here---------------end--------------->8---

For the subset that we did package, we were always trailing the
Linux-libre releases, so the argument that waiting for their releases
would hamper our security doesn't seem to hold.

> also make it prohibitively difficult to perform deblobbed bisections
> between two adjacent versions from the upstream stable git repository.

In my opinion, we should not trade our correctness guarantee in exchange
for convenience, especially if the convenience is only gained in such a
corner case as per-commit bisection of the Linux kernel.  It'd be
oversimplifying to say that the Linux-libre developers just run their
scripts to produce a release; they also manually screen the new upstream
changes and update their scripts accordingly.  To give due credit to
their efforts, we should not simply run their scripts with a newer
version/commit of Linux and expect arriving at a correct result.

> In my opinion, at minimum, the 'linux-libre-upstream-source' argument to
> 'make-linux-libre-source' should optional.

Perhaps, like for the change proposed by Leo, the edge case of bisecting
per-commit could be accommodated by reverting this patch when needed?
It seems more important that the common case be rigorously verified.

Also note that it should be possible to:

1) Test each packaged release in Guix to "bisect" (duh)
2) Test any Linux stable release via the Linux-libre git repo, building
with a command such as "guix build
--with-git-url=linux-libre=git://linux-libre.fsfla.org/releases.git
--with-commit=linux-libre=sources/v5.8.3-gnu linux-libre". Unfortunately
this can't be done from the command line using 'guix system build ...'
but it should be easy to define your own linux-libre package using the
'make-linux-libre*' procedure (which will gladly accept any linux-libre
source).

For when the per-commit granularity is not required.

In the future, the linux-libre git repo will apply their clean ups per
commit, allowing to do like 2) above for any commit.

> I find it depressing that Jason's and Alexandre's attempts to browbeat
> us to limit ourselves to deblob only the precise tarballs that they
> produce, and to always wait for them to produce them before pushing
> security fixes (although it takes less than 10 minutes to look over the
> upstream commits for new blobs) have gained traction here.

Despite the somewhat corrosive tone of the exchange, some valid points
were made.  I've scavenged these and adapted the recipe.  I think the
end result is a win-win situation for both Linux-libre and Guix.

As shown above, there hasn't been a case where the Linux-libre effort
slowed down the deployment of a new Linux kernel version in Guix.  I
don't foresee this changing.

What do you think? Are there holes in my analysis/understanding?

Thank you,

Maxim