Are gzip-compressed substitutes still used?

[Ludovic Courtès]

Hi Nicolò,

Nicolò Balzarotti <anothersms@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> We could also drop gzip, but there are probably pre-1.1 daemons out
>> there that understand nothing but gzip¹, so perhaps that’ll have to
>> wait.  Now, compressing substitutes three times may be somewhat
>> unreasonable.
>>
>> Thoughts?
>>
> Is there a request log where we can check whether this is true?

I finally got around to checking this.

I picked a relatively popular substitute for which the lzip-compressed
variant is smaller than the gzip-compressed variant, and thus modern
‘guix substitute’ chooses lzip over gzip:

--8<---------------cut here---------------start------------->8---
$ wget -q -O - https://ci.guix.gnu.org/7rpj4dmn9g64zqp8vkc0byx93glix2pm.narinfo 
| head -7
StorePath: /gnu/store/7rpj4dmn9g64zqp8vkc0byx93glix2pm-gtk+-3.24.23
URL: nar/gzip/7rpj4dmn9g64zqp8vkc0byx93glix2pm-gtk%2B-3.24.23
Compression: gzip
FileSize: 13982949
URL: nar/lzip/7rpj4dmn9g64zqp8vkc0byx93glix2pm-gtk%2B-3.24.23
Compression: lzip
FileSize: 7223862
--8<---------------cut here---------------end--------------->8---

On berlin, I looked at the HTTPS nginx logs and did this:

--8<---------------cut here---------------start------------->8---
ludo@berlin /var/log/nginx$ tail -10000000 < https.access.log > /tmp/sample.log
ludo@berlin /var/log/nginx$ date
Thu 28 Jan 2021 06:18:01 PM CET
ludo@berlin /var/log/nginx$ grep /7rpj4dmn9g64zqp8vkc0byx93glix2pm-gtk < 
/tmp/sample.log |wc -l
1304
ludo@berlin /var/log/nginx$ grep /gzip/7rpj4dmn9g64zqp8vkc0byx93glix2pm-gtk < 
/tmp/sample.log |wc -l
17
ludo@berlin /var/log/nginx$ grep /lzip/7rpj4dmn9g64zqp8vkc0byx93glix2pm-gtk < 
/tmp/sample.log |wc -l
1287
--8<---------------cut here---------------end--------------->8---

The 10M-request sample covers requests from Jan. 10th to now.  Over that
period, 99% of the GTK+ downloads were made as lzip.  We see similar
results with less popular packages and with core packages:

--8<---------------cut here---------------start------------->8---
ludo@berlin /var/log/nginx$ grep /01xi3sig314wgwa1j9sxk37vl816mj74-r-minimal < 
/tmp/sample.log | wc -l
85
ludo@berlin /var/log/nginx$ grep 
/gzip/01xi3sig314wgwa1j9sxk37vl816mj74-r-minimal < /tmp/sample.log | wc -l
1
ludo@berlin /var/log/nginx$ grep 
/lzip/01xi3sig314wgwa1j9sxk37vl816mj74-r-minimal < /tmp/sample.log | wc -l
84
ludo@berlin /var/log/nginx$ grep /0m0vd873jp61lcm4xa3ljdgx381qa782-guile-3.0.2 
< /tmp/sample.log |wc -l
1601
ludo@berlin /var/log/nginx$ grep 
/gzip/0m0vd873jp61lcm4xa3ljdgx381qa782-guile-3.0.2 < /tmp/sample.log |wc -l
8
ludo@berlin /var/log/nginx$ grep 
/lzip/0m0vd873jp61lcm4xa3ljdgx381qa782-guile-3.0.2 < /tmp/sample.log |wc -l
1593
--8<---------------cut here---------------end--------------->8---

>From that, we could deduce that about 1% of our users who take
substitutes from ci.guix are still using a pre-1.1.0 daemon without
support for lzip compression.

I find it surprisingly low: 1.1.0 was released “only” 9 months ago,
which is not a lot for someone used to the long release cycles of
“stable” distros.

It might be underestimated: users running an old daemon probably update
less often and may thus be underrepresented in the substitute logs.


As for whether it’s OK to drop gzip substitutes altogether: I’m not
confident about knowingly breaking 1% or more of the deployed Guixes,
but it’s all about tradeoffs.

Ludo’.