[bug#59078] [PATCH] lint: Split the derivation lint checker by system.

[Ludovic Courtès]

Hi,

Christopher Baines <mail@cbaines.net> skribis:

> Currently, if you attempt to run the derivation checker on all packages, the
> Guile process will run out of memory. I think a contributing factor to this is
> that the checker forces an inefficient order when you want to generate
> derivations for all the supported systems of each package, constantly
> switching system then package.
>
> This problem also impacts the Guix Data Service, since it tries to run the
> derivation checker for all packages.
>
> The changes in this commit to split the derivation lint checker in to several,
> one for each system, means that you can now treat each system separately,
> which should be better for caching purposes.
>
> If it's desirable to keep some notion of checking all supported systems for a
> single package, I think lint checker groups could be added, so that you could
> ask for the "derivation" checker, and this would run all the derivation
> checkers.

The ‘derivation’ checker was added for this purpose: making sure that a
package’s derivation can be computed for all the supported systems.
Previously it was easy to overlook that kind of breakage.

I think it’s important to keep a ‘derivation’ checker that does this.


Now, the memory consumption you report is unacceptable and this needs to
be addressed.

Most (all?) caches are now per-session (associated with
<store-connection>).  Since ‘guix lint’ uses a single session, those
caches keep growing because there’s no eviction mechanism in place.

A hack like the one below should work around that.  Could you check how
well it works for you? It can also be helpful to set:

  GUIX_PROFILING=gc GUIX_PROFILING_EVENTS=after-gc

Longer-term we should have a proper cache eviction mechanism in place.
It’s not been a priority because “normal” applications such as ‘guix
package’ don’t need it.

Thanks,
Ludo’.

diff --git a/guix/scripts/lint.scm b/guix/scripts/lint.scm
index 9920c3ee62..6d9b9e96a9 100644
--- a/guix/scripts/lint.scm
+++ b/guix/scripts/lint.scm
@@ -209,23 +209,31 @@ (define (parse-options)
       (list-checkers-and-exit checkers))
 
     (with-error-handling
-      (let ((any-lint-checker-requires-store?
-             (any lint-checker-requires-store? checkers)))
+      (let ((store-needed? (any lint-checker-requires-store? checkers)))
 
-        (define (call-maybe-with-store proc)
-          (if any-lint-checker-requires-store?
-              (with-store store
-                (proc store))
-              (proc #f)))
-
-        (call-maybe-with-store
-         (lambda (store)
-           (cond
-            ((null? args)
-             (fold-packages (lambda (p r) (run-checkers p checkers
-                                                        #:store store)) '()))
-            (else
-             (for-each (lambda (package)
-                         (run-checkers package checkers
-                                       #:store store))
-                       args)))))))))
+        (cond
+         ((null? args)
+          (let loop ((packages  (fold-packages cons '()))
+                     (processed 0)
+                     (store     #f))
+            (match packages
+              ((package . rest)
+               (let ((store (and store-needed?
+                                 (if store
+                                     ;; XXX: Periodically start a new session
+                                     ;; with empty caches to avoid unbounded
+                                     ;; heap growth caused by the 'derivation'
+                                     ;; checker.
+                                     (if (zero? (modulo processed 1000))
+                                         (begin
+                                           (close-connection store)
+                                           (open-connection))
+                                         store)
+                                     (open-connection)))))
+                 (run-checkers package checkers #:store store)
+                 (loop rest (+ 1 processed) store))))))
+         (else
+          (for-each (lambda (package)
+                      (run-checkers package checkers
+                                    #:store store))
+                    args)))))))