bug#55721: Low cache usage in the presence of grafts

[Efraim Flashner]

On Mon, May 30, 2022 at 11:31:47AM +0200, Ludovic Courtès wrote:
> Ludovic Courtès <ludo@gnu.org> skribis:
> 
> > $ time GUIX_PROFILING="rpc reference-cache derivation-graft-cache" 
> > ./pre-inst-env  guix system vm gnu/system/examples/desktop.tmpl 
> > 0.2 MB will be downloaded
> >  polkit-0.121  198KiB                                                       
> >                          1.4MiB/s 00:00 [##################] 100.0%
> > /gnu/store/sk1571rnh5jl2ilp0v1k8bqwsdrngwxy-run-vm.sh
> > Reference Cache:
> >   fresh caches:     8
> >   lookups:      77581
> >   hits:         52890 (68.2%)
> >   cache size:     838 entries
> > Derivation graft cache:
> >   fresh caches:    39
> >   lookups:      704635
> >   hits:         627819 (89.1%)
> >   cache size:    2946 entries
> > Remote procedure call summary: 33071 RPCs
> >   built-in-builders              ...     1
> >   query-substitutable-path-infos ...     2
> >   build-things                   ...     2
> >   add-to-store/tree              ...    56
> >   valid-path?                    ...    59
> >   add-to-store                   ...   373
> >   add-text-to-store              ...  7887
> >   query-references               ... 24691
> >
> > real        5m6.727s
> > user        6m59.438s
> > sys 0m1.633s
> 
> With this patch:
> 

> diff --git a/guix/store.scm b/guix/store.scm
> index efba07bdcd..88cab1bb6a 100644
> --- a/guix/store.scm
> +++ b/guix/store.scm
> @@ -1337,7 +1337,12 @@ (define (build-accumulator expected-store)
>      (if (and (eq? (store-connection-socket store)
>                    (store-connection-socket expected-store))
>               (= mode (build-mode normal)))
> -        (unresolved things continue)
> +        (unresolved things
> +                    (lambda (new-store value)
> +                      ;; Borrow caches from NEW-STORE.
> +                      (set-store-connection-caches!
> +                       store (store-connection-caches new-store))
> +                      (continue value)))
>          (continue #t))))
>  
>  (define default-cutoff
> @@ -1397,7 +1402,8 @@ (define-values (result rest)
>                                        (if (unresolved? obj)
>                                            ;; Pass #f because 'build-things' 
> is now
>                                            ;; unnecessary.
> -                                          ((unresolved-continuation obj) #f)
> +                                          ((unresolved-continuation obj)
> +                                           store #f)
>                                            obj))
>                                      result #:cutoff cutoff)
>           (map/accumulate-builds store proc rest #:cutoff cutoff)))))

> 
> … we’re doing better:
> 
> --8<---------------cut here---------------start------------->8---
> $ guix gc -D $(guix build polkit --no-grafts)
> finding garbage collector roots...
> [0 MiB] deleting '/gnu/store/8vsfk8312m5p461wl4dq5lxjagfbwfhz-polkit-0.121'
> deleting `/gnu/store/trash'
> deleting unused links...
> note: currently hard linking saves 72397.13 MiB
> $ time GUIX_PROFILING="rpc reference-cache derivation-graft-cache" 
> ./pre-inst-env  guix system vm gnu/system/examples/desktop.tmpl 
> 0.2 MB will be downloaded
>  polkit-0.121  198KiB                                                         
>                        1.6MiB/s 00:00 [##################] 100.0%
> /gnu/store/wp3lv4xrh6vw79gnkyi5471c1l8j624n-run-vm.sh
> Reference Cache:
>   fresh caches:     8
>   lookups:      17232
>   hits:         14816 (86.0%)
>   cache size:     784 entries
> Derivation graft cache:
>   fresh caches:    39
>   lookups:      128402
>   hits:         111336 (86.7%)
>   cache size:     328 entries
> Remote procedure call summary: 10796 RPCs
>   built-in-builders              ...     1
>   query-substitutable-path-infos ...     2
>   build-things                   ...     2
>   add-to-store/tree              ...    56
>   valid-path?                    ...    59
>   add-to-store                   ...   373
>   query-references               ...  2416
>   add-text-to-store              ...  7887
> 
> real  1m31.550s
> user  2m7.107s
> sys   0m0.637s
> --8<---------------cut here---------------end--------------->8---
> 
> There’s still room for improvement though.
> 
> Ludo’.

With that change you've dropped the query-references from 24691 to 2416,
more than a 90% drop. If you want to take aim at improving it further
I'd say take a look at add-text-to-store and see if you can shrink that
number.

-- 
Efraim Flashner   <efraim@flashner.co.il>   אפרים פלשנר
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted

signature.asc
Description: PGP signature