Re: Performance of computing cross derivations

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> When asked by the data service, it seems to take Guix around 3 minutes
> to compute cross derivations for all packages (to a single
> target). Here's a simple script that replicates this:

To understand the cost of computing a package’s derivation, I generally
start looking at caches and memoization:

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING="object-cache" guix build gcc-toolchain -d --no-grafts
/gnu/store/iwn6frqqcyw808sgsnjv26dn6rq7mijd-gcc-toolchain-13.2.0.drv
Object Cache:
  fresh caches:    19
  lookups:       3667
  hits:          3342 (91.1%)
  cache size:     323 entries
$ GUIX_PROFILING="object-cache" guix build sed -d --no-grafts 
--target=aarch64-linux-gnu
/gnu/store/yxakl87wizwzcqapx4sdkp56652cxb4m-sed-4.8.drv
Object Cache:
  fresh caches:    20
  lookups:       5420
  hits:          4919 (90.8%)
  cache size:     500 entries
--8<---------------cut here---------------end--------------->8---

Caches are critical: since we’re dealing with huge package graphs, we
need to make sure we don’t end up computing the same thing several
times.  (You can also add “memoization” to the ‘GUIX_PROFILING’ variable
above.)

One idiom that defeats caching is:

  (define (make-me-a-package x y z)
    (package
      …))

Such a procedure returns a fresh package every time it’s called,
preventing caching from happening (because cache entries are compared
with ‘eq?’).  That typically leads to lower hit rates.

Anyway, lots of words to say that I don’t see anything immediately
obvious with cross-compilation, yet I wouldn’t be surprised if some of
these cache-defeating idioms were used because we’ve payed less
attention to this.

An even better thing to start with: compare the timing of ‘guix build -d
--no-grafts $PKG --target=aarch64-linux-gnu’ for all valid values of
$PKG, and investigate those that take the most time.

HTH!

Ludo’.