Re: When substitute download + decompression is CPU-bound

[Guillaume Le Vaillant]

I compared gzip, lzip and zstd when compressing a 580 MB pack (therefore
containing "subsitutes" for several packages) with different compression
levels. Maybe the results can be of some use to someone.

Note that the plots only show the results using only 1 thread and
standard compression levels, and that the speed axis is using
logarithmic scale.

Machine used for the tests:
 - CPU: Intel i7-3630QM
 - RAM: 16 MiB

Programs:
 - gzip 1.10
 - pigz 2.4
 - lzip 1.21
 - plzip 1.8
 - zstd 1.4.4
 - pzstd 1.4.4

Uncompressed file:
 - name: monero-0.17.1.5-pack.tar
 - size: 582707200 bytes

#+PLOT: script:"compression-benchmark.plot"
| Comp. command  | Comp. time | Comp. size | Comp. speed | Comp. ratio | 
Decomp. time | Decomp. speed |
|----------------+------------+------------+-------------+-------------+--------------+---------------|
| gzip -1        |      7.999 |  166904534 |    72847506 |       3.491 |        
3.292 |      50700041 |
| gzip -2        |      8.469 |  161859128 |    68804723 |       3.600 |        
3.214 |      50360650 |
| gzip -3        |     10.239 |  157839772 |    56910558 |       3.692 |        
3.144 |      50203490 |
| gzip -4        |     11.035 |  151039457 |    52805365 |       3.858 |        
3.104 |      48659619 |
| gzip -5        |     13.767 |  146693142 |    42326375 |       3.972 |        
3.143 |      46672969 |
| gzip -6        |     19.707 |  144364588 |    29568539 |       4.036 |        
3.001 |      48105494 |
| gzip -7        |     24.014 |  143727357 |    24265312 |       4.054 |        
2.993 |      48021168 |
| gzip -8        |     43.219 |  143062985 |    13482663 |       4.073 |        
2.969 |      48185579 |
| gzip -9        |     70.930 |  142803637 |     8215243 |       4.080 |        
2.964 |      48179365 |
| pigz -1 -p 4   |      2.247 |  165745308 |   259326747 |       3.516 |        
1.919 |      86370666 |
| pigz -2 -p 4   |      2.394 |  160661935 |   243403175 |       3.627 |        
1.862 |      86284605 |
| pigz -3 -p 4   |      2.776 |  156696382 |   209908934 |       3.719 |        
1.817 |      86239065 |
| pigz -4 -p 4   |      3.045 |  150539955 |   191365255 |       3.871 |        
1.787 |      84241721 |
| pigz -5 -p 4   |      3.855 |  146289903 |   151156213 |       3.983 |        
1.732 |      84462992 |
| pigz -6 -p 4   |      5.378 |  143967093 |   108350167 |       4.048 |        
1.721 |      83653163 |
| pigz -7 -p 4   |      6.579 |  143350506 |    88570786 |       4.065 |        
1.702 |      84224739 |
| pigz -8 -p 4   |      11.76 |  142738270 |    49549932 |       4.082 |        
1.720 |      82987366 |
| pigz -9 -p 4   |     19.878 |  142479078 |    29314176 |       4.090 |        
1.691 |      84257290 |
| lzip -0        |     16.686 |  130302649 |    34921923 |       4.472 |        
9.981 |      13055070 |
| lzip -1        |     42.011 |  118070414 |    13870348 |       4.935 |        
8.669 |      13619842 |
| lzip -2        |     51.395 |  112769303 |    11337819 |       5.167 |        
8.368 |      13476255 |
| lzip -3        |     69.344 |  106182860 |     8403138 |       5.488 |        
8.162 |      13009417 |
| lzip -4        |     89.781 |  100072461 |     6490318 |       5.823 |        
7.837 |      12769231 |
| lzip -5        |    119.626 |   95033235 |     4871075 |       6.132 |        
7.586 |      12527450 |
| lzip -6        |    155.740 |   83063613 |     3741538 |       7.015 |        
6.856 |      12115463 |
| lzip -7        |    197.485 |   78596381 |     2950640 |       7.414 |        
6.586 |      11933857 |
| lzip -8        |    238.076 |   72885403 |     2447568 |       7.995 |        
6.227 |      11704738 |
| lzip -9        |    306.368 |   72279340 |     1901985 |       8.062 |        
6.203 |      11652320 |
| plzip -0 -n 4  |      4.821 |  131211238 |   120868533 |       4.441 |        
2.829 |      46380784 |
| plzip -1 -n 4  |     13.453 |  120565830 |    43314294 |       4.833 |        
2.604 |      46300242 |
| plzip -2 -n 4  |     15.695 |  114874773 |    37126932 |       5.073 |        
2.398 |      47904409 |
| plzip -3 -n 4  |     20.563 |  108896468 |    28337655 |       5.351 |        
2.486 |      43803889 |
| plzip -4 -n 4  |     26.871 |  102285879 |    21685356 |       5.697 |        
2.375 |      43067739 |
| plzip -5 -n 4  |     35.220 |   97402840 |    16544781 |       5.982 |        
2.448 |      39788742 |
| plzip -6 -n 4  |     45.812 |   89260273 |    12719532 |       6.528 |        
2.145 |      41613181 |
| plzip -7 -n 4  |     62.723 |   82944080 |     9290168 |       7.025 |        
2.080 |      39876962 |
| plzip -8 -n 4  |     71.928 |   78477272 |     8101257 |       7.425 |        
2.120 |      37017581 |
| plzip -9 -n 4  |    103.744 |   75648923 |     5616780 |       7.703 |        
2.578 |      29344035 |
| zstd -1        |      2.057 |  145784609 |   283280117 |       3.997 |        
0.639 |     228144928 |
| zstd -2        |      2.316 |  136049621 |   251600691 |       4.283 |        
0.657 |     207077049 |
| zstd -3        |      2.733 |  127702753 |   213211562 |       4.563 |        
0.650 |     196465774 |
| zstd -4        |      3.269 |  126224007 |   178252432 |       4.616 |        
0.658 |     191829798 |
| zstd -5        |      5.136 |  122024478 |   113455452 |       4.775 |        
0.680 |     179447762 |
| zstd -6        |      6.394 |  120035201 |    91133438 |       4.854 |        
0.652 |     184103069 |
| zstd -7        |      8.510 |  116048780 |    68473231 |       5.021 |        
0.612 |     189622190 |
| zstd -8        |      9.875 |  114821611 |    59008324 |       5.075 |        
0.593 |     193628349 |
| zstd -9        |     12.478 |  113868149 |    46698766 |       5.117 |        
0.588 |     193653315 |
| zstd -10       |     14.982 |  111113753 |    38893819 |       5.244 |        
0.578 |     192238327 |
| zstd -11       |     16.391 |  110674252 |    35550436 |       5.265 |        
0.583 |     189835767 |
| zstd -12       |     21.008 |  110031164 |    27737395 |       5.296 |        
0.570 |     193037130 |
| zstd -13       |     51.259 |  109262475 |    11367900 |       5.333 |        
0.561 |     194763770 |
| zstd -14       |     58.897 |  108632734 |     9893665 |       5.364 |        
0.562 |     193296680 |
| zstd -15       |     82.514 |  107956132 |     7061919 |       5.398 |        
0.557 |     193817113 |
| zstd -16       |     78.935 |  105533404 |     7382114 |       5.522 |        
0.576 |     183217715 |
| zstd -17       |     89.832 |   94165409 |     6486633 |       6.188 |        
0.565 |     166664441 |
| zstd -18       |    115.663 |   91124039 |     5037974 |       6.395 |        
0.614 |     148410487 |
| zstd -19       |    157.008 |   90229137 |     3711322 |       6.458 |        
0.614 |     146952992 |
| zstd -20       |    162.499 |   80742922 |     3585913 |       7.217 |        
0.605 |     133459375 |
| zstd -21       |    207.122 |   79619348 |     2813353 |       7.319 |        
0.611 |     130309899 |
| zstd -22       |    277.177 |   78652901 |     2102293 |       7.409 |        
0.634 |     124058203 |
| pzstd -1 -p 4  |      0.621 |  146665510 |   938336876 |       3.973 |        
0.196 |     748293418 |
| pzstd -2 -p 4  |      0.720 |  137416958 |   809315556 |       4.240 |        
0.227 |     605361048 |
| pzstd -3 -p 4  |      1.180 |  128748806 |   493819661 |       4.526 |        
0.231 |     557354139 |
| pzstd -4 -p 4  |      1.786 |  127373154 |   326263830 |       4.575 |        
0.240 |     530721475 |
| pzstd -5 -p 4  |      2.635 |  123216422 |   221141252 |       4.729 |        
0.240 |     513401758 |
| pzstd -6 -p 4  |      3.774 |  121257316 |   154400424 |       4.806 |        
0.251 |     483096876 |
| pzstd -7 -p 4  |      3.988 |  117361187 |   146115145 |       4.965 |        
0.263 |     446240255 |
| pzstd -8 -p 4  |      4.540 |  116172098 |   128349604 |       5.016 |        
0.240 |     484050408 |
| pzstd -9 -p 4  |      5.083 |  115237287 |   114638442 |       5.057 |        
0.268 |     429989877 |
| pzstd -10 -p 4 |      5.630 |  112359994 |   103500391 |       5.186 |        
0.226 |     497168115 |
| pzstd -11 -p 4 |      5.991 |  111969711 |    97263762 |       5.204 |        
0.246 |     455161427 |
| pzstd -12 -p 4 |      8.001 |  111326376 |    72829296 |       5.234 |        
0.227 |     490424564 |
| pzstd -13 -p 4 |     16.035 |  110525395 |    36339707 |       5.272 |        
0.259 |     426738977 |
| pzstd -14 -p 4 |     18.145 |  109957500 |    32113927 |       5.299 |        
0.253 |     434614625 |
| pzstd -15 -p 4 |     24.791 |  109358520 |    23504788 |       5.328 |        
0.224 |     488207679 |
| pzstd -16 -p 4 |     23.940 |  106888588 |    24340317 |       5.452 |        
0.234 |     456788838 |
| pzstd -17 -p 4 |     29.099 |   97393935 |    20024991 |       5.983 |        
0.266 |     366142613 |
| pzstd -18 -p 4 |     37.124 |   94273955 |    15696240 |       6.181 |        
0.284 |     331950546 |
| pzstd -19 -p 4 |     48.798 |   93531545 |    11941211 |       6.230 |        
0.262 |     356990630 |
| pzstd -20 -p 4 |     54.860 |   82067608 |    10621713 |       7.100 |        
0.302 |     271747046 |
| pzstd -21 -p 4 |     64.179 |   79735488 |     9079406 |       7.308 |        
0.389 |     204975548 |
| pzstd -22 -p 4 |    256.242 |   78688788 |     2274050 |       7.405 |        
0.585 |     134510749 |
#+TBLFM: $4='(format "%d" (round (/ 582707200.0 $2)));N :: $5='(format "%.3f" 
(/ 582707200.0 $3));N :: $7='(format "%d" (round (/ $3 $6)));N

compression-benchmark.plot
Description: Text document

compression.png
Description: PNG image

decompression.png
Description: PNG image

signature.asc
Description: PGP signature