[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: When substitute download + decompression is CPU-bound
|
From: |
Nicolò Balzarotti |
|
Subject: |
Re: When substitute download + decompression is CPU-bound |
|
Date: |
Fri, 29 Jan 2021 15:55:49 +0100 |
Pierre Neidhardt <mail@ambrevar.xyz> writes:
> Nicolò Balzarotti <anothersms@gmail.com> writes:
>
> What wouldn't be the case? If you mean that "gzip is never the best
> choice", wouldn't Zstd outperform gzip on the Raspberry Pi 1 too?
My bad, you are right. Also, memory usage shoudn't be a problem. gzip
uses way less (testd on ungoogled chromium, I get ~16kb peak heap size
for gzip, 8Mb for zstd and 32Mb for lzip), but I'd expect guix to be
running on systems with more than 8Mb of memory. Just for reference,
here's the memory profiling script
#+PROPERTY: header-args:bash :session valgrind
* NAR Decompression memory benchmark
#+begin_src bash :results none
guix environment --ad-hoc valgrind lzip gzip zstd wget
#+end_src
#+begin_src bash :cache yes
valgrind --version | sed 's/-/ /'
lzip --version | head -1
gunzip --version | head -1 | sed 's/\s(/(/'
zstd --version | sed -e 's/command.*v//' -e 's/,.*//' -e 's/**//'
#+end_src
#+RESULTS[e07cecfd5cc770b7a898408b80678f2e8ea7772e]:
| valgrind | 3.16.1 |
| lzip | 1.21 |
| gunzip(gzip) | 1.1 |
| zstd | 1.4.4 |
Just noticed that there should be a new zstd release
([[https://github.com/facebook/zstd/releases/][zstd 1.4.8]]), and
a new lzip release ([[https://download.savannah.gnu.org/releases/lzip/][lzip
1.22]]).
** Prepare required data
#+begin_src bash :cache yes
wget
https://ci.guix.gnu.org/nar/gzip/kfcrrl6p6f6v51jg5rirmq3q067zxih6-ungoogled-chromium-87.0.4280.88-0.b78cb92
-O uc.nar.gz
wget
https://ci.guix.gnu.org/nar/lzip/kfcrrl6p6f6v51jg5rirmq3q067zxih6-ungoogled-chromium-87.0.4280.88-0.b78cb92
-O uc.nar.lz
#+end_src
#+RESULTS[ea17e5a54da1ca54a9c82f264912675d9ca981a0]:
Create zstd compressed file
#+begin_src bash :results none
gunzip -c < uc.nar.gz > uc.nar
zstd -19 uc.nar -o uc.nar.zstd
#+end_src
Check file sizes
#+begin_src bash
ls -lh --sort=size | head -5
#+end_src
#+RESULTS:
| total | 585M | | | | | | | |
| -rw-r--r-- | 1 | nixo | users | 335M | Jan | 29 | 15:14 | uc.nar |
| -rw-r--r-- | 1 | nixo | users | 103M | Jan | 29 | 15:13 | uc.nar.gz |
| -rw-r--r-- | 1 | nixo | users | 78M | Jan | 29 | 15:14 | uc.nar.zstd |
| -rw-r--r-- | 1 | nixo | users | 71M | Jan | 29 | 15:13 | uc.nar.lz |
** Decompress
#+name: massif
#+begin_src bash :session valgrind :var command="ls" input="."
output="/dev/null" name="ls"
time valgrind --tool=massif --log-file=/dev/null --time-unit=B
--trace-children=yes --massif-out-file=$name.massif $command < $input >$output
#+end_src
#+call: massif(command="gunzip -c", input="uc.nar.gz", output="/dev/null",
name="gzip")
#+RESULTS:
| nixo@guixSD | ~/prof | [env]$ | nixo@guixSD | ~/prof | [env]$ |
nixo@guixSD | ~/prof | [env]$ |
| real | 0m8.291s | | | | |
| | |
| user | 0m7.910s | | | | |
| | |
| sys | 0m0.201s | | | | |
| | |
#+call: massif(command="lzip -d", input="uc.nar.lz", output="/dev/null",
name="lzip")
#+RESULTS:
| nixo@guixSD | ~/prof | [env]$ | nixo@guixSD | ~/prof | [env]$ |
nixo@guixSD | ~/prof | [env]$ |
| real | 0m22.378s | | | | |
| | |
| user | 0m20.959s | | | | |
| | |
| sys | 0m0.345s | | | | |
| | |
#+call: massif(command="zstd -d", input="uc.nar.zstd", output="/dev/null",
name="zstd")
#+RESULTS:
| nixo@guixSD | ~/prof | [env]$ | nixo@guixSD | ~/prof | [env]$ |
nixo@guixSD | ~/prof | [env]$ |
| real | 0m4.607s | | | | |
| | |
| user | 0m4.157s | | | | |
| | |
| sys | 0m0.135s | | | | |
| | |
** Check massif output
#+begin_src bash :results raw drawer
for ext in gzip lzip zstd; do
ms_print $ext.massif > $ext.graph
done
#+end_src
#+RESULTS:
:results:
:end:
--------------------------------------------------------------------------------
Command:
/gnu/store/378zjf2kgajcfd7mfr98jn5xyc5wa3qv-gzip-1.10/bin/gzip -d -c
Massif arguments: --time-unit=B --massif-out-file=gzip.massif
ms_print arguments: gzip.massif
--------------------------------------------------------------------------------
KB
15.59^ #
| #
| #
| #
| #
| # :
| # :
| # :
| : # :: : :
| : # :: : : :
| : :#: ::::: :: : :: : : : : : @@ :
| : :#: :: : :: : : : : : : : @ :
| : :#: :: : :: : : : : : : : @ :
| : :#: :: : :: : : : : : : : : @ :
| : :#: :: : :: : : : : : : : : @ :
| : :#: :: : ::: ::: :::: : : : :: : @ : :
| : :#: :: : ::: ::: :: :: : : : :::: :::: : @ : :
| : :#::::: : ::: ::: : :: : ::: : ::: : ::: :::@@ : @ ::: :
| : :#:: :: : :::::::: : :: : :: :: ::: : ::: :: @ :::@ :: :
| ::::#:: :: : :::: ::: : :: : :: :: ::: : ::: :: @ :::@ :: : :
0 +----------------------------------------------------------------------->MB
0 114.8
Number of snapshots: 53
Detailed snapshots: [5 (peak), 21, 39, 42, 46, 50]
--------------------------------------------------------------------------------
Command: lzip -d
Massif arguments: --time-unit=B --massif-out-file=lzip.massif
ms_print arguments: lzip.massif
--------------------------------------------------------------------------------
MB
32.09^ ###################################
| #
| #
| #
| #
| #
| #
| #
| #
| #
| #
| #
| #
| #
| #
| #
| #
| #
| #
| #
0 +----------------------------------------------------------------------->MB
0 64.18
Number of snapshots: 12
Detailed snapshots: [6 (peak)]
--------------------------------------------------------------------------------
Command: zstd -d
Massif arguments: --time-unit=B --massif-out-file=zstd.massif
ms_print arguments: zstd.massif
--------------------------------------------------------------------------------
MB
8.665^ #
| :#:::::::::::::::::::::::::::::::::
| :#
| :#
| :#
| :#
| :#
| :#
| :#
| :#
| :#
| :#
| :#
| :#
| :#
| :#
| :#
| :#
| :#
| :#
0 +----------------------------------------------------------------------->MB
0 17.33
Number of snapshots: 18
Detailed snapshots: [9 (peak)]
- Re: When substitute download + decompression is CPU-bound, (continued)
- Re: When substitute download + decompression is CPU-bound, Ludovic Courtès, 2021/01/14
- Re: When substitute download + decompression is CPU-bound, Nicolò Balzarotti, 2021/01/14
- Re: When substitute download + decompression is CPU-bound, Pierre Neidhardt, 2021/01/15
- Re: When substitute download + decompression is CPU-bound, Ludovic Courtès, 2021/01/28
- Re: When substitute download + decompression is CPU-bound, Pierre Neidhardt, 2021/01/29
- Re: When substitute download + decompression is CPU-bound, Guillaume Le Vaillant, 2021/01/29
- Re: When substitute download + decompression is CPU-bound, Nicolò Balzarotti, 2021/01/29
- Re: When substitute download + decompression is CPU-bound, Pierre Neidhardt, 2021/01/29
- Re: When substitute download + decompression is CPU-bound, Guillaume Le Vaillant, 2021/01/29
- Re: When substitute download + decompression is CPU-bound,
Nicolò Balzarotti <=
- Re: When substitute download + decompression is CPU-bound, zimoun, 2021/01/29