[bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances

[zimoun]

Hi Ludo,

On Fri, 23 Jul 2021 at 17:43, Ludovic Courtès <ludo@gnu.org> wrote:

> > From my understanding, the issue that 'package-relevance' accepts a 
> > 'package'
> > (and then all the chain until displaying) and 'fold-avaibale-packages' does
> > not return a package.  Well, I do not know; especially where to put 
> > something
> > similiar to 'read-package-from'.
>
> Yeah that’s annoying.  Perhaps we need <proto-package> or
> <package-metadata>.  With some trickery we could have record type
> inheritance or something, maybe.  Dunno.
>
> It would be good if we could arrange so that ‘fold-available-packages’
> doesn’t allocate anything though.

It does not allocate, the allocation is done by 'find-packages-by-description'.

Well, I think v4/v3 [0] is the direction to follow.  Therefore, I
would revisit this version and try to address two of Ludo's comments
[1] and the other ones in v3.

BTW, I have not investigated from where the slowness comes:
 - allocation
 - garbage collection
 - '(module-ref (resolve-interface module) symbol)'
because I have been a bit disappointed by the performance of this v6.

0: <http://issues.guix.gnu.org/39258#89>
1: <http://issues.guix.gnu.org/39258#93>

> > Let compare only for cold cache and time this cache building (Guix 7db8fd6):
> >
> >   sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
> >   time guix build --check $(guix gc --derivers $(readlink -f 
> > ~/.config/guix/current/lib/guix/package.cache))
> >
> >   real        0m28,848s
> >   user        0m1,481s
> >   sys 0m0,252s
> >
> >   sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
> >   time guix build --check $(guix gc --derivers $(readlink -f 
> > /tmp/new/lib/guix/package.cache))
> >
> >   real        0m40,279s
> >   user        0m1,582s
> >   sys 0m0,232s
> >
> > It seems longer but compared to the time of "guix pull" completion, it seems
> > acceptable.
>
> Both the initial timing and the target are waaay too much.  :-/

Yes, but that's another story. :-)  We cannot fix all in the same time, IMHO.

Here the point is to speedup "guix search" and to accept to pay a
little more at "guix pull" time; then we could optimize the cache
generation.  Considering the overall time of "guix pull", this extra
time appears to me acceptable---if "guix search" is faster. :-)

> On my i7 laptop I have:
>
> --8<---------------cut here---------------start------------->8---
> $ time ./pre-inst-env  guile -c '(use-modules (gnu packages)) 
> (generate-package-cache "/tmp/t.cache")'
>
> real    0m20.738s
> user    0m44.413s
> sys     0m0.341s
> --8<---------------cut here---------------end--------------->8---
>
> It’s CPU-bound; we should probably start by optimizing that.
>
> In Guile 3.0.7 there was a change that improved this noticeably:
>
>   
> https://git.savannah.gnu.org/cgit/guile.git/commit/?id=05614f792bfabbc33798863edd0bb67c744e9299
>
> We should prolly look for similar optimization opportunities in the
> assembler…

Yes, but this kind of optimization will not provide a faster "guix
search" but a faster "guix pull".

--8<---------------cut here---------------start------------->8---
$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ time guile -c '(use-modules (gnu packages)) (generate-package-cache
"/tmp/t1.cache")'

real    0m15,728s
user    0m23,940s
sys     0m0,826s

$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ time guile -c '(load-compiled "/tmp/t1.cache/lib/guix/package.cache")'

real    0m1,026s
user    0m0,258s
sys     0m0,051s

$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ time ./pre-inst-env  guile -c '(use-modules (gnu packages))
(generate-package-cache "/tmp/t2.cache")'

real    0m35,570s
user    3m12,951s
sys     0m3,807s

$ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
$ time guile -c '(load-compiled "/tmp/t2.cache/lib/guix/package.cache")'

real    0m1,055s
user    0m0,283s
sys     0m0,055s
--8<---------------cut here---------------end--------------->8---

(And if we speak about performance, raw loading the cache takes ~1s
but "guix package -A >/dev/null" takes ~8s.  It is a big gap for
parsing the CLI and sorting.  Worse, "guix --version >/dev/null" takes
~2s on cold cache.  We should probably start by reduce this overhead.)

> > Let compare for some queries:
> >
> >   sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
> >   time guix search game | recsel -C -P name | wc -l
> >   371
> >
> >   real        0m7,561s
> >   user        0m3,525s
> >   sys 0m0,391s
>
> I think you should run:
>
>   time guix search game > /dev/null
>
> otherwise Bash’s built-in ‘time’ shows the wall-clock time of the whole
> pipeline, and the processing time of ‘recsel’ is probably not negligible
> here.

First, I am confused... If the formatter (recsel) is not negligible,
then it should be dropped and replaced by an internal (fast)
formatter.  Well, I mean that as an end-user I am interested by the
time of the whole pipeline.

Second, on my machine the time is somehow negligible*. ;-)  On cold
cache, 10 runs using the pipe or using the redirection, keeping the
max and the min for each:

--8<---------------cut here---------------start------------->8---
real    0m9,598s
user    0m3,961s
sys     0m0,415s

real    0m8,744s
user    0m3,772s
sys     0m0,431s
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
real    0m8,755s
user    0m3,869s
sys     0m0,540s

real    0m8,390s
user    0m3,767s
sys     0m0,416s
--8<---------------cut here---------------end--------------->8---

*negligible: better said, it does not give the bad impression.  Even
if it is roughly 5% of difference.


Cheers,
simon