[bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances

[zimoun]

Hi,

This is an attempt to improve the performance of "guix search".  It is still
half baked but it allows to discuss further the idea about expanding the
current '/lib/guix/package.cache' and avoids to forget an IRL discussion. ;-)

Let start by what needs to be improved: the part when cache is not
authoritative.  It is slower than the current approach because the package is
read twice, i.e., the module is indeed loaded twice, once by
'fold-available-packages' via 'fold-module-public-variables*' and then again
by 'find-packages-by-description' via 'read-package-from'.  The issue is to
have a common interface for both cases (cache and no-cache).  More thoughts
are required. ;-)

Then, using the cache is slower than expected.  Therefore, something is maybe
twisted -- quick implementation before holidays ;-) -- with the use of
'fold-avaibale-packages' as proposed by Ludo [1].  Note that instead another
'fold-packages' (say 'fold-packages*') using the new cache should be used.  As
it is done with v4 and the performances were as expected:

   <http://issues.guix.gnu.org/39258#89>

1: <http://issues.guix.gnu.org/39258#93>

>From my understanding, the issue that 'package-relevance' accepts a 'package'
(and then all the chain until displaying) and 'fold-avaibale-packages' does
not return a package.  Well, I do not know; especially where to put something
similiar to 'read-package-from'.


To test, after applying the patches, the command is:

   ./pre-inst-env guix pull --allow-downgrades --disable-authentication \
          --url=$(pwd) --branch=search-v6 -p /tmp/new


Let compare only for cold cache and time this cache building (Guix 7db8fd6):

  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time guix build --check $(guix gc --derivers $(readlink -f 
~/.config/guix/current/lib/guix/package.cache))

  real  0m28,848s
  user  0m1,481s
  sys   0m0,252s

  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time guix build --check $(guix gc --derivers $(readlink -f 
/tmp/new/lib/guix/package.cache))

  real  0m40,279s
  user  0m1,582s
  sys   0m0,232s

It seems longer but compared to the time of "guix pull" completion, it seems
acceptable.  However, maybe it could become an issue when running a lot of
"guix time-machine"... Well, hard trade-off. ;-)

Let compare for some queries:

--8<---------------cut here---------------start------------->8---
  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time guix search game | recsel -C -P name | wc -l
  371

  real  0m7,561s
  user  0m3,525s
  sys   0m0,391s

  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time /tmp/new/bin/guix search game | recsel -C -P name | wc -l
  371

  real  0m9,814s
  user  0m3,240s
  sys   0m0,363s
--8<---------------cut here---------------end--------------->8---


  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time guix search strategy game | recsel -C -P name | wc -l
  16

  real  0m8,565s
  user  0m2,803s
  sys   0m0,430s

  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time /tmp/new/bin/guix search strategy game | recsel -C -P name | wc -l
  16

  real  0m9,679s
  user  0m2,370s
  sys   0m0,334s


--8<---------------cut here---------------start------------->8---
  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time guix search strategy game caesar | recsel -C -P name | wc -l
  0

  real  0m8,307s
  user  0m2,388s
  sys   0m0,366s

  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time /tmp/new/bin/guix search strategy game caesar | recsel -C -P name | wc -l
  0

  real  0m3,626s
  user  0m0,948s
  sys   0m0,101s
--8<---------------cut here---------------end--------------->8---


  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time guix search game strategy the | recsel -C -P name | wc -l
  4

  real  0m8,776s
  user  0m2,903s
  sys   0m0,454s

  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time /tmp/new/bin/guix search game strategy the | recsel -C -P name | wc -l
  4

  real  0m9,495s
  user  0m2,546s
  sys   0m0,313s


--8<---------------cut here---------------start------------->8---
  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time guix search the game strategy | recsel -C -P name | wc -l
  4

  real  0m8,502s
  user  0m2,534s
  sys   0m0,388s

  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time /tmp/new/bin/guix search the game strategy | recsel -C -P name | wc -l
  4

  real  0m9,508s
  user  0m2,254s
  sys   0m0,363s
--8<---------------cut here---------------end--------------->8---


  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time guix search crypto library | recsel -C -P name | grep libb2
  libb2

  real  0m8,744s
  user  0m2,875s
  sys   0m0,374s

  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time /tmp/new/bin/guix search crypto library | recsel -C -P name | grep libb2
  libb2

  real  0m9,229s
  user  0m2,448s
  sys   0m0,397s


--8<---------------cut here---------------start------------->8---
  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time guix search cuirass integration | recsel -C -P name | wc -l
  1

  real  0m8,132s
  user  0m2,343s
  sys   0m0,407s

  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time /tmp/new/bin/guix search cuirass integration | recsel -C -P name | wc -l
  1

  real  0m8,940s
  user  0m2,036s
  sys   0m0,369s
--8<---------------cut here---------------end--------------->8---


  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time guix search cuirass | recsel -C -P name | wc -l
  2

  real  0m8,240s
  user  0m2,461s
  sys   0m0,367s

  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time /tmp/new/bin/guix search cuirass | recsel -C -P name | wc -l
  2

  real  0m8,863s
  user  0m2,019s
  sys   0m0,377s


--8<---------------cut here---------------start------------->8---
  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time guix search cuirass integration foo | recsel -C -P name | wc -l
  0

  real  0m8,258s
  user  0m2,418s
  sys   0m0,521s

  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time /tmp/new/bin/guix search cuirass integration foo | recsel -C -P name | 
wc -l
  0

  real  0m3,358s
  user  0m0,867s
  sys   0m0,139s
--8<---------------cut here---------------end--------------->8---

This last example suggests that 'read-package-from' is the slowdown.


(On a side note, maybe I am doing wrong, but there is no improvement by the
recent introduction of 'cut' for multi-terms as the query "the game strategy"
and "game strategy the".  Another story. :-))


When cache is not authoritative, it is worse, as expected:

  sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
  time /tmp/new/bin/guix search -L /tmp/my-pkgs cuirass integration foo | 
recsel -C -P name | wc -l
  0

  real  0m12,503s
  user  0m7,807s
  sys   0m0,529s

and note that currently the performances of "guix search" is the same for both
cases (authoritative and not authoritative); i.e., see previous timing.


Last, two minor remarks about previous comments.

1. Ludo commented:

        > Therefore the cache '/lib/guix/package.cache' contains more
        > information.

        This breaks the binary interface, so we’ll have to analyze the impact of
        such a change and devise a strategy.

        <http://issues.guix.gnu.org/39258#93>

and after some checking, this should be fine, IIUC.  The '--news' is ok
because of '#:allow-other-keys'.  And other parts are modified accordingly.
Guix revision N creates a cache that Guix revision N+1 will read but it should
not be an issue; see 'inferior-available-packages'.

2. And Ludo wrote:

        I realize the other cache also has that problem, but it would be nice 
to add a
        version tag to the cache.  Basically emit something like:

          (package-metadata-cache (version 0) VECTOR …)

        instead of just:

          (VECTOR …)

        <http://issues.guix.gnu.org/39258#93>

which is, after discussions, not necessary.  Versioning does not make sense
here because the cache is read by the Guix which generates it.  Therefore,
specify a version is extraneous here.


Comments are welcome for this work-in-progress. ;-)

Cheers,
simon


zimoun (2): DRAFT packages: Add fields to packages cache.  DRAFT scripts:
  package: Use cache in 'find-packages-by-description'.

 gnu/packages.scm         | 52 +++++++++++++++++++++++++++-------------
 guix/scripts/package.scm | 46 +++++++++++++++++++++++++----------
 2 files changed, 70 insertions(+), 28 deletions(-)


base-commit: 4196087f3d6fc254db5b4c47658e5679c835516f
-- 
2.32.0