bug#55721: Low cache usage in the presence of grafts

[Ludovic Courtès]

Ludovic Courtès <ludo@gnu.org> skribis:

> $ time GUIX_PROFILING="rpc reference-cache derivation-graft-cache" 
> ./pre-inst-env  guix system vm gnu/system/examples/desktop.tmpl 
> 0.2 MB will be downloaded
>  polkit-0.121  198KiB                                                         
>                        1.4MiB/s 00:00 [##################] 100.0%
> /gnu/store/sk1571rnh5jl2ilp0v1k8bqwsdrngwxy-run-vm.sh
> Reference Cache:
>   fresh caches:     8
>   lookups:      77581
>   hits:         52890 (68.2%)
>   cache size:     838 entries
> Derivation graft cache:
>   fresh caches:    39
>   lookups:      704635
>   hits:         627819 (89.1%)
>   cache size:    2946 entries
> Remote procedure call summary: 33071 RPCs
>   built-in-builders              ...     1
>   query-substitutable-path-infos ...     2
>   build-things                   ...     2
>   add-to-store/tree              ...    56
>   valid-path?                    ...    59
>   add-to-store                   ...   373
>   add-text-to-store              ...  7887
>   query-references               ... 24691
>
> real  5m6.727s
> user  6m59.438s
> sys   0m1.633s

With this patch:

diff --git a/guix/store.scm b/guix/store.scm
index efba07bdcd..88cab1bb6a 100644
--- a/guix/store.scm
+++ b/guix/store.scm
@@ -1337,7 +1337,12 @@ (define (build-accumulator expected-store)
     (if (and (eq? (store-connection-socket store)
                   (store-connection-socket expected-store))
              (= mode (build-mode normal)))
-        (unresolved things continue)
+        (unresolved things
+                    (lambda (new-store value)
+                      ;; Borrow caches from NEW-STORE.
+                      (set-store-connection-caches!
+                       store (store-connection-caches new-store))
+                      (continue value)))
         (continue #t))))
 
 (define default-cutoff
@@ -1397,7 +1402,8 @@ (define-values (result rest)
                                       (if (unresolved? obj)
                                           ;; Pass #f because 'build-things' is 
now
                                           ;; unnecessary.
-                                          ((unresolved-continuation obj) #f)
+                                          ((unresolved-continuation obj)
+                                           store #f)
                                           obj))
                                     result #:cutoff cutoff)
          (map/accumulate-builds store proc rest #:cutoff cutoff)))))

… we’re doing better:

--8<---------------cut here---------------start------------->8---
$ guix gc -D $(guix build polkit --no-grafts)
finding garbage collector roots...
[0 MiB] deleting '/gnu/store/8vsfk8312m5p461wl4dq5lxjagfbwfhz-polkit-0.121'
deleting `/gnu/store/trash'
deleting unused links...
note: currently hard linking saves 72397.13 MiB
$ time GUIX_PROFILING="rpc reference-cache derivation-graft-cache" 
./pre-inst-env  guix system vm gnu/system/examples/desktop.tmpl 
0.2 MB will be downloaded
 polkit-0.121  198KiB                                                           
                     1.6MiB/s 00:00 [##################] 100.0%
/gnu/store/wp3lv4xrh6vw79gnkyi5471c1l8j624n-run-vm.sh
Reference Cache:
  fresh caches:     8
  lookups:      17232
  hits:         14816 (86.0%)
  cache size:     784 entries
Derivation graft cache:
  fresh caches:    39
  lookups:      128402
  hits:         111336 (86.7%)
  cache size:     328 entries
Remote procedure call summary: 10796 RPCs
  built-in-builders              ...     1
  query-substitutable-path-infos ...     2
  build-things                   ...     2
  add-to-store/tree              ...    56
  valid-path?                    ...    59
  add-to-store                   ...   373
  query-references               ...  2416
  add-text-to-store              ...  7887

real    1m31.550s
user    2m7.107s
sys     0m0.637s
--8<---------------cut here---------------end--------------->8---

There’s still room for improvement though.

Ludo’.