bug#44735: gilbc of the running system got wiped while building a package, system broken

[Stefan]

Hi!

After trying to just build a package (with a modified guix, but this is 
certainly unrelated), the system broke catastrophically:

stefan@guix ~/development/guix$ sudo -E ./pre-inst-env guix-daemon 
--build-users-group=guixbuild &
stefan@guix ~/development/guix$ /home/stefan/development/guix/pre-inst-env guix 
build -L /home/stefan/guix u-boot-rpi-3
accepted connection from pid 23848, user stefan
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
12,5 MB will be downloaded:
   /gnu/store/ldg4jqfan2vp01lm255zz7zrb4vllixp-libxau-1.0.9
   /gnu/store/m1r4jwmc56q44x31xcnvg1hcijf0lq88-libxcb-1.14
   /gnu/store/8b75zmsyxc5qghfrxhyqi6g23bq993b1-libbsd-0.10.0
   /gnu/store/z18hwxwgk551y4a0f6j1dxhmp208i4ha-bash-static-5.0.16
   /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31
   /gnu/store/z7hanmdmdalqh1v0y7z8ilinfhyfh91d-glibc-2.31-static
   /gnu/store/pcsl88vd66k62sk1g4wcc9i985xn369m-libxdmcp-1.1.3
   /gnu/store/785ldh00ix897pamyg5p6fpjls6ddwzz-libx11-1.6.A-doc
   /gnu/store/x10mk7ri4ny013km57d3h5093270r7pg-libx11-1.6.A
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substituting /gnu/store/785ldh00ix897pamyg5p6fpjls6ddwzz-libx11-1.6.A-doc...
downloading from 
https://ci.guix.gnu.org/nar/lzip/785ldh00ix897pamyg5p6fpjls6ddwzz-libx11-1.6.A-doc
 ...
 libx11-1.6.A-doc  1.2MiB                                          309KiB/s 
00:04 [##################] 100.0%

substituting /gnu/store/z18hwxwgk551y4a0f6j1dxhmp208i4ha-bash-static-5.0.16...
downloading from 
https://ci.guix.gnu.org/nar/lzip/z18hwxwgk551y4a0f6j1dxhmp208i4ha-bash-static-5.0.16
 ...
 bash-static-5.0.16  502KiB                                        976KiB/s 
00:01 [##################] 100.0%

guix build: error: cannot unlink 
`/gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/share/locale/de/LC_MESSAGES':
 Directory not empty
stefan@guix ~/development/guix$ /home/stefan/development/guix/pre-inst-env guix 
build -L /home/stefan/guix u-boot-rpi-3
accepted connection from pid 23911, user stefan
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
12,0 MB will be downloaded:
   /gnu/store/ldg4jqfan2vp01lm255zz7zrb4vllixp-libxau-1.0.9
   /gnu/store/m1r4jwmc56q44x31xcnvg1hcijf0lq88-libxcb-1.14
   /gnu/store/8b75zmsyxc5qghfrxhyqi6g23bq993b1-libbsd-0.10.0
   /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31
   /gnu/store/z7hanmdmdalqh1v0y7z8ilinfhyfh91d-glibc-2.31-static
   /gnu/store/pcsl88vd66k62sk1g4wcc9i985xn369m-libxdmcp-1.1.3
   /gnu/store/785ldh00ix897pamyg5p6fpjls6ddwzz-libx11-1.6.A-doc
   /gnu/store/x10mk7ri4ny013km57d3h5093270r7pg-libx11-1.6.A
guix build: error: cannot unlink 
`/gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib/gconv': Directory 
not empty
stefan@guix ~/development/guix$ /home/stefan/development/guix/pre-inst-env guix 
build -L /home/stefan/guix u-boot-rpi-3
-bash: /home/stefan/development/guix/pre-inst-env: /bin/sh: Defekter 
Interpreter: No such file or directory
stefan@guix ~/development/guix$ ls
-bash: /run/current-system/profile/bin/ls: No such file or directory
stefan@guix ~/development/guix$ echo $PATH
/run/setuid-programs:/home/stefan/.config/guix/current/bin:/home/stefan/.guix-profile/bin:/run/current-system/profile/bin:/run/current-system/profile/sbin
stefan@guix ~/development/guix$ ls
-bash: /run/current-system/profile/bin/ls: No such file or directory


The problem that the unlink was not successful was certainly due to a deleted 
but still opened file on the NFS share. There may be an intermediate hidden 
.nfs… file, which get created in such a case (“delete on last close”, “silly 
rename”), However, the RFC-5661 for NFS demands even if 
OPEN4_RESULT_PRESERVE_UNLINKED is supported, that the directory entry of an 
open file must not be removed in this case, thus preventing a directory removal.

Taking a look on the nfs-server, this is now the content of the glibc:

# ls gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31/lib -l
total 12
drwxr-xr-x 2 root root 12288 Nov 18 19:41 gconv
lrwxrwxrwx 7 root root    11 Jan  1  1970 libanl.so -> libanl.so.1

Everything is missing.

Of course I tried to reboot the system, but because of the missing 
ld-linux-aarch64.so.1 the system is not booting properly.


This leaves some questions:

If the whole system (maybe not the booted/running one, but a reconfigured one) 
was already using /gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31, why 
has it been downloaded again when building a package? Was this due to grafting?

How can it be that the store is kind of “state“ being modified on the fly?

How does it come that removing files for 
/gnu/store/cb88z63hyg1icd2kkahiink2p291mhr2-glibc-2.31 happens before the 
download even started? Could this render a system unusable if a network loss 
happens at the right point in time?

Booting the previous system generation luckily worked – it must be using a 
different glibc version. But now what about that broken glibc version? How to 
repair it? The garbage collector will not remove it, as it is still referenced 
by the latest system generation. Actually I don’t want to delete that 
generation. The database certainly believes, that this glibc package is 
installed correctly. What to do now?

The used guix version used when building the package was 
f6a42ac946edccc7de5e93ee247487cbec40072b.


Bye

Stefan