bug#38062: Offloading + timeout + --keep-going leads to assertion failure

[Ludovic Courtès]

Hello,

An offloaded build that times out in the presence of ‘--keep-going’
leads to an assertion failure.  To reproduce, set up offloading and run
something like this:

--8<---------------cut here---------------start------------->8---
$ guix build vim --no-substitutes --timeout=5 --keep-going
The following derivation will be built:
   /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv
process 4277 acquired build slot '/var/guix/offload/localhost:2222/0'
load on machine 'localhost' is 0.04 (normalized: 0.01)
building /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv...
sending 1 store item (48 MiB) to 'localhost'...
exporting path 
`/gnu/store/mlwyk5vcja0gqm20xxj8mwgf0fbqv8cz-vim-8.1.0644-checkout'
building of `/gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv' 
timed out after 5 seconds
build of /gnu/store/5mnnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv failed
View build log at 
'/var/log/guix/drvs/5m/nnym9xwl145s1b88aqfmrka810w9ci-vim-8.1.0644.drv.bz2'.
guix build: error: corrupt input while restoring archive from #<closed: file 
7ff2769cdc40>
--8<---------------cut here---------------end--------------->8---

The last error is due to premature EOF on the client socket: the child
‘guix-daemon’ process crashed, and thus ‘guix build’ gets EPIPE on its
client socket.  Here’s how the ‘guix-daemon’ process crashed:

--8<---------------cut here---------------start------------->8---
$ tail -3 /var/log/guix-daemon.log
accepted connection from pid 4270, user ludo
accepted connection from pid 4277, user root
guix-daemon: nix/libstore/build.cc:3448: void nix::Worker::run(const Goals&): 
Assertion `!settings.keepGoing || children.empty()' failed.
--8<---------------cut here---------------end--------------->8---

In Cuirass, this manifests itself with EPIPE while writing to the client
socket:

--8<---------------cut here---------------start------------->8---
2019-11-04T20:11:26 fatal: uncaught exception 'system-error' in 
'restart-builds' fiber!
2019-11-04T20:11:26 exception arguments: ("fport_write" "~A" ("Broken pipe") 
(32))
In ice-9/boot-9.scm:
    829:9  5 (catch _ _ #<procedure 7f79d663c320 at cuirass/utils.s?> ?)
   751:25  4 (dispatch-exception 0 system-error ("fport_write" "~A" ?))
In cuirass/utils.scm:
    183:8  3 (_ _ "fport_write" "~A" ("Broken pipe") (32))
In ice-9/boot-9.scm:
    829:9  2 (catch #t #<procedure 7f79cc7e41c0 at cuirass/utils.sc?> ?)
In cuirass/utils.scm:
   184:22  1 (_)
In unknown file:
           0 (make-stack #t)
ERROR: In procedure make-stack:
In procedure fport_write: Broken pipe
--8<---------------cut here---------------end--------------->8---

This bug is almost certainly a consequence of commit
ada9a19a2dca74feafcf24df1152abd685d4142f.

To be continued…

Ludo’.