[bug#44899] [PATCH 0/3] Using 'ld.so.cache' to speed up application startup

[Ludovic Courtès]

Hello Guix!

The other day on IRC Ricardo had the brilliant idea of using the
ld.so cache to avoid the “stat storm” stemming from our long RUNPATHs,
and thus to speed up application startup.  As an example, Guile has
9 entries in its RUNPATH and Inkscape has 44 entries.

The first patch changes the loader (1) to look for the cache in
$ORIGIN/../etc/ld.so.cache, and (2) to look for the cache before
looking at RUNPATH entries.

The second patch adds a build phase that creates ‘etc/ld.so.cache’.
It passes ‘ldconfig’ a config file that contains the union of all
the RUNPATH entries of all the executables found in the output at
hand.  (It cannot be done in a profile hook because $ORIGIN is
determined by looking at /proc/self/exe, which is the canonical file
name in the store.)

You can see it in action with LD_DEBUG=libs:

--8<---------------cut here---------------start------------->8---
$ LD_DEBUG=libs 
/gnu/store/3dfv892jq8081xnsl9gfylbgv1fdicvd-guile-3.0.4/bin/guile --version
     11150:     find library=libguile-3.0.so.1 [0]; searching
     11150:      search 
cache=/gnu/store/3dfv892jq8081xnsl9gfylbgv1fdicvd-guile-3.0.4/bin/../etc/ld.so.cache
     11150:       trying 
file=/gnu/store/3dfv892jq8081xnsl9gfylbgv1fdicvd-guile-3.0.4/lib/libguile-3.0.so.1
     11150:
     11150:     find library=libgc.so.1 [0]; searching
     11150:      search 
cache=/gnu/store/3dfv892jq8081xnsl9gfylbgv1fdicvd-guile-3.0.4/bin/../etc/ld.so.cache
     11150:       trying 
file=/gnu/store/hy88vf2ynlica0wj0ppi0d3b11gi2b2h-libgc-8.0.4/lib/libgc.so.1

[...]
--8<---------------cut here---------------end--------------->8---

Here’s the after/before for Guile:

--8<---------------cut here---------------start------------->8---
$ strace -c /gnu/store/3dfv892jq8081xnsl9gfylbgv1fdicvd-guile-3.0.4/bin/guile 
--version

[...]

  2.70    0.000259           5        46         6 openat
  1.87    0.000180           1       130        88 stat
  1.66    0.000159           4        36           rt_sigprocmask
  0.74    0.000071           1        40           close
  0.64    0.000061           3        18           fstat

[...]

100.00    0.009604                   600       105 total
$ strace -c guile --version

[...]

 13.82    0.000723           4       165       114 openat
 13.46    0.000704           3       190       144 stat

[...]

  1.43    0.000075           2        29           fstat

[...]

100.00    0.005232                   773       268 total
--8<---------------cut here---------------end--------------->8---

Erroneous syscalls are divided by 2.5; total syscalls reduced by 22%.

For Bash:

--8<---------------cut here---------------start------------->8---
$ strace -c 
/gnu/store/qs33sf58502v1wx77va092y14sbspv4f-bash-minimal-5.0.16/bin/bash 
--version
GNU bash, version 5.0.16(1)-release (x86_64-unknown-linux-gnu)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  0.00    0.000000           0         4           read
  0.00    0.000000           0         6           write
  0.00    0.000000           0         5           close
  0.00    0.000000           0         5           fstat
  0.00    0.000000           0        12           mmap
  0.00    0.000000           0         5           mprotect
  0.00    0.000000           0         1           munmap
  0.00    0.000000           0         3           brk
  0.00    0.000000           0         1           rt_sigprocmask
  0.00    0.000000           0         1         1 access
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         1           readlink
  0.00    0.000000           0         1           getuid
  0.00    0.000000           0         1           getgid
  0.00    0.000000           0         1           geteuid
  0.00    0.000000           0         1           getegid
  0.00    0.000000           0         1           arch_prctl
  0.00    0.000000           0         9         4 openat
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                    59         5 total
$ strace -c 
/gnu/store/fvhj74pghapbjvsvj27skvkra1by1965-bash-minimal-5.0.16/bin/bash 
--version
GNU bash, version 5.0.16(1)-release (x86_64-unknown-linux-gnu)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 34.15    0.000420          12        35        16 openat
 18.29    0.000225           8        27           mmap
  8.13    0.000100          25         4           brk
  7.72    0.000095           6        14        14 stat
  7.24    0.000089          89         1         1 access
  6.18    0.000076           4        19           fstat
  5.45    0.000067           3        19           close
  4.47    0.000055           2        20           read
  3.82    0.000047           9         5           mprotect
  2.36    0.000029           4         6           write
  0.65    0.000008           8         1           execve
  0.41    0.000005           5         1           arch_prctl
  0.33    0.000004           4         1           getuid
  0.24    0.000003           3         1           rt_sigprocmask
  0.24    0.000003           3         1           getegid
  0.16    0.000002           2         1           getgid
  0.16    0.000002           2         1           geteuid
------ ----------- ----------- --------- --------- ----------------
100.00    0.001230                   157        31 total
--8<---------------cut here---------------end--------------->8---

Erroneous syscalls are divided by 6; total syscalls divided by 2.7.

As always, this is probably not that big a deal on warm-cache SSD,
but it probably makes a difference on a cold cache, on spinning
disks, and on network file systems.

* Possible improvements

The hard-coded ‘../etc/ld.so.cache’ means that it can only be used
with first-level sub-directories like bin/ and sbin/; it won’t be
used for libexec/guix/guile, for instance, which is a bummer.
Perhaps we should compute the ‘ld.so.cache’ file name “lexically”
instead.

We should also think hard about ways users could be tricked into
loading a malicious ‘ld.so.cache’.  That’s also another reason why
“lexical dot-dot” would be safer: we could ensure that only
pre-computed ‘ld.so.cache’ that live in the store are ever loaded.

The ‘ld.so.conf’ file passed to ‘ldconfig’ should ideally contains
the RUNPATH entries _recursively_, such that even indirect
dependencies can be found in cache.

Thoughts?

Ludo’.

Ludovic Courtès (3):
  gnu: glibc: Load ${ORIGIN}/../etc/ld.so.cache when available.
  gremlin: Fix typo in docstring.
  build-system/gnu: Add 'make-dynamic-linker-cache' phase.

 gnu/local.mk                              |   1 +
 gnu/packages/base.scm                     |  11 +-
 gnu/packages/patches/glibc-dl-cache.patch | 122 ++++++++++++++++++++++
 guix/build-system/gnu.scm                 |   4 +
 guix/build/gnu-build-system.scm           |  73 +++++++++++++
 guix/build/gremlin.scm                    |   2 +-
 6 files changed, 202 insertions(+), 11 deletions(-)
 create mode 100644 gnu/packages/patches/glibc-dl-cache.patch

-- 
2.29.2