Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

[Julien Moutinho]

Bash Version: 5.1
Patch Level: 8
Release Status: release
Architecture: x86_64-linux

Description:

bash-5.1 reaches crashing code paths
when launched by systemd-249 or valgrind.
I cannot get such crashes when bash is built using:
./configure --without-bash-malloc

Repeat-By:

Here's a Nix recipe for reproducible builds,
but you should be able to reproduce the crash with your own build method.

$ cat flake.nix
> {
> inputs.nixpkgs.url = 
> "github:NixOS/nixpkgs/62ca4a425ab112e5d8a75682ee76c069cdb42d4b";
> outputs = inputs:
>   let system = "x86_64-linux";
>       pkgs = inputs.nixpkgs.legacyPackages.${system};
>   in
>   {
>     packages.${system} = {
>       bash5-with-bash-malloc = (pkgs.bash
>         .override { stdenv = pkgs.keepDebugInfo pkgs.stdenv; })
>         .overrideAttrs (old: {
>           configureFlags = old.configureFlags ++ ["--with-bash-malloc"];
>           CFLAGS="-DDEBUG -DMALLOC_DEBUG";
>       });
>       bash5-without-bash-malloc = (pkgs.bash
>         .override { stdenv = pkgs.keepDebugInfo pkgs.stdenv; })
>         .overrideAttrs (old: {
>           configureFlags = old.configureFlags ++ ["--without-bash-malloc"];
>           CFLAGS="-DDEBUG -DMALLOC_DEBUG";
>       });
>       bash4-with-bash-malloc = (pkgs.bash_4
>         .override { stdenv = pkgs.keepDebugInfo pkgs.stdenv; })
>         .overrideAttrs (old: {
>           configureFlags = old.configureFlags ++ ["--with-bash-malloc"];
>           CFLAGS="-DDEBUG -DMALLOC_DEBUG";
>       });
>       bash4-without-bash-malloc = (pkgs.bash_4
>         .override { stdenv = pkgs.keepDebugInfo pkgs.stdenv; })
>         .overrideAttrs (old: {
>           configureFlags = old.configureFlags ++ ["--without-bash-malloc"];
>           CFLAGS="-DDEBUG -DMALLOC_DEBUG";
>       });
>     };
>   };
> }

See 
https://github.com/NixOS/nixpkgs/blob/62ca4a425ab112e5d8a75682ee76c069cdb42d4b/pkgs/shells/bash/5.1.nix
for the full bash-5.1 build recipe of pkgs.bash,
that this flake.nix's overrides to enable debug symbols and switch bash's 
malloc.

$ nix eval .#bash5-with-bash-malloc.name
> "bash-5.1-p8"

$ nix -L build .#bash5-with-bash-malloc
$ file result/bin/bash
> result/bin/bash: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), 
> dynamically linked, interpreter 
> /nix/store/2jfn3d7vyj7h0h6lmh510rz31db68l1i-glibc-2.33-50/lib/ld-linux-x86-64.so.2,
>  for GNU/Linux 2.6.32, with debug_info, not stripped

$ sudo mkdir -p /run/bash
$ sudo systemd-run --pipe -p DynamicUser=1 -E LD_LIBRARY_PATH=$(nix-store -q 
$(which systemctl))/lib -pBindReadOnlyPaths={/etc,/nix}  -p RootDirectory=/opt 
-- $(readlink -e result/bin/bash) --norc
> Running as unit: run-u534.service
> 
> malloc: unknown:0: assertion botched
> malloc: 0x37c8fa10: allocated: last allocated from unknown:0
> realloc: start and end chunk sizes differ
> last command: (null)
> Aborting...

Note that DynamicUser=1 runs bash using a temporary user
supplied by the libnss_systemd.so.2 found at the given LD_LIBRARY_PATH=
$ grep passwd: /etc/nsswitch.conf 
> passwd:    files systemd

Not also that the proposed flake.nix does not provision systemd-run,
the systemd used here is:
$ systemctl --version
> systemd 249 (249.4)
> +PAM +AUDIT -SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL 
> +ACL +BLKID +CURL -ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP 
> +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB -ZSTD 
> -XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified

Debugging:

$ sudo coredumpctl debug
>            PID: 229855 (bash)
>            UID: 65483 (65483)
>            GID: 65483 (65483)
>         Signal: 6 (ABRT)
>      Timestamp: Mon 2021-10-04 04:09:08 CEST (3s ago)
>   Command Line: 
> /nix/store/acnfapqyd414vrzhhgclgqn37ll0rdr3-bash-5.1-p8/bin/bash --norc
>     Executable: 
> /nix/store/acnfapqyd414vrzhhgclgqn37ll0rdr3-bash-5.1-p8/bin/bash
>  Control Group: /system.slice/run-u534.service
>           Unit: run-u534.service
>          Slice: system.slice
>        Boot ID: 8f9583b087c14b5c813501bbccb1ef11
>     Machine ID: d434cb63c21e4dd193a3203059583bb3
>       Hostname: losurdo
>        Storage: 
> /var/lib/systemd/coredump/core.bash.65483.8f9583b087c14b5c813501bbccb1ef11.229855.1633313348000000.lz4
>  (present)
>      Disk Size: 69.2K
>        Message: Process 229855 (bash) of user 65483 dumped core.
> 
> GNU gdb (GDB) 10.2
> Copyright (C) 2021 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> Type "show copying" and "show warranty" for details.
> This GDB was configured as "x86_64-unknown-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <https://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
>     <http://www.gnu.org/software/gdb/documentation/>.
> 
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from 
> /nix/store/acnfapqyd414vrzhhgclgqn37ll0rdr3-bash-5.1-p8/bin/bash...
> [New LWP 229855]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library 
> "/nix/store/2jfn3d7vyj7h0h6lmh510rz31db68l1i-glibc-2.33-50/lib/libthread_db.so.1".
> Core was generated by 
> `/nix/store/acnfapqyd414vrzhhgclgqn37ll0rdr3-bash-5.1-p8/bin/bash --norc'.
> Program terminated with signal SIGABRT, Aborted.
> #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
> 49      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
> (gdb) bt
> #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
> #1  0x0000619871e455d3 in __GI_abort () at abort.c:100
> #2  0x0000000000440d49 in programming_error (format=format@entry=0x4b9708 
> "realloc: start and end chunk sizes differ")
>     at error.c:175
> #3  0x000000000049e408 in xbotch (mem=mem@entry=0x37c8fa10, e=e@entry=8, 
>     s=0x4b9708 "realloc: start and end chunk sizes differ", 
> file=file@entry=0x0, line=line@entry=0) at malloc.c:376
> #4  0x000000000049f5c3 in internal_realloc (mem=0x37c8fa10, n=218, 
> file=file@entry=0x0, line=line@entry=0, 
>     flags=flags@entry=0) at malloc.c:1150
> #5  0x000000000049f867 in realloc (mem=<optimized out>, nbytes=<optimized 
> out>) at malloc.c:1440
> #6  0x0000619871c02526 in greedy_realloc ()
>    from 
> /nix/store/5iw38x4l9nyp2srnlxfi9l1q9np0pchq-systemd-249.4/lib/libnss_systemd.so.2
> #7  0x0000619871c04c44 in read_line_full ()
>    from 
> /nix/store/5iw38x4l9nyp2srnlxfi9l1q9np0pchq-systemd-249.4/lib/libnss_systemd.so.2
> #8  0x0000619871c04eb6 in read_one_line_file ()
>    from 
> /nix/store/5iw38x4l9nyp2srnlxfi9l1q9np0pchq-systemd-249.4/lib/libnss_systemd.so.2
> #9  0x0000619871bfd1b8 in proc_cmdline_parse ()
>    from 
> /nix/store/5iw38x4l9nyp2srnlxfi9l1q9np0pchq-systemd-249.4/lib/libnss_systemd.so.2
> #10 0x0000619871bfa939 in log_parse_environment ()
>    from 
> /nix/store/5iw38x4l9nyp2srnlxfi9l1q9np0pchq-systemd-249.4/lib/libnss_systemd.so.2
> #11 0x0000619871b965a7 in __pthread_once_slow (once_control=0x619871c490a4 
> <once>, 
>     init_routine=0x619871bf3b20 <setup_logging>) at pthread_once.c:116
> #12 0x0000619871bf40ff in _nss_systemd_getpwuid_r ()
>    from 
> /nix/store/5iw38x4l9nyp2srnlxfi9l1q9np0pchq-systemd-249.4/lib/libnss_systemd.so.2
> #13 0x0000619871ee7b46 in __getpwuid_r (uid=uid@entry=65483, 
> resbuf=resbuf@entry=0x619871fdbf40 <resbuf>, 
>     buffer=<optimized out>, buflen=buflen@entry=1024, 
> result=result@entry=0x7b3c23e18658) at ../nss/getXXbyYY_r.c:274
> #14 0x0000619871ee7378 in getpwuid (uid=65483) at ../nss/getXXbyYY.c:135
> #15 0x000000000041c9d2 in get_current_user_info () at shell.c:1869
> #16 0x000000000041cc8a in shell_initialize () at shell.c:1932
> #17 0x000000000041d1e8 in main (argc=2, argv=0x7b3c23e18918, 
> env=0x7b3c23e18930) at shell.c:572

Note the _nss_systemd_getpwuid_r in the backtrace.

AFAICS:
- No crash happens with bash5-with-bash-malloc on systemd-247.
- No crash happens with bash5-without-bash-malloc on systemd-247 or systemd-249.
- A slightly different crash, still involving _nss_systemd_getpwuid_r,
  happens with bash4-with-bash-malloc on systemd-247.
  It might have been fixed when a new definition of MALIGN_MASK
  was introduced in bash-5.1:
  
https://git.savannah.gnu.org/cgit/bash.git/commit/lib/malloc/malloc.c?id=8868edaf2250e09c4e9a1c75ffe3274f28f38581
  See also: https://www.mail-archive.com/bug-bash@gnu.org/msg24306.html
- No crash happens with bash4-without-bash-malloc on systemd-247 or systemd-249.
- bash crashes inside valgrind too,
  but apparently something different is happening
  because it crashes even without systemd being involved:

$ nix build .#bash5-with-bash-malloc
$ valgrind result/bin/bash --norc -c true
> ==307088== Memcheck, a memory error detector
> ==307088== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> ==307088== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
> ==307088== Command: result/bin/bash --norc -c true
> ==307088== 
> ==307088== Invalid free() / delete / delete[] / realloc()
> ==307088==    at 0x483F8E9: free (in 
> /nix/store/7s7hzqaf5imxmpjlxh2n6fs7ixml98ya-valgrind-3.16.1/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==307088==    by 0x47330F: xfree (xmalloc.c:150)
> ==307088==    by 0x4644FA: unwind_frame_run_internal (unwind_prot.c:325)
> ==307088==    by 0x4640B6: without_interrupts (unwind_prot.c:117)
> ==307088==    by 0x464656: run_unwind_frame (unwind_prot.c:143)
> ==307088==    by 0x479ACA: parse_and_execute (evalstring.c:523)
> ==307088==    by 0x41C0A5: run_one_command (shell.c:1440)
> ==307088==    by 0x41D6A1: main (shell.c:741)
> ==307088==  Address 0x404be10 is in the brk data segment 0x4033000-0x4054fff
> ==307088== 
> ==307088== 
> ==307088== HEAP SUMMARY:
> ==307088==     in use at exit: 0 bytes in 0 blocks
> ==307088==   total heap usage: 2,410 allocs, 2,411 frees, 156,759 bytes 
> allocated
> ==307088== 
> ==307088== All heap blocks were freed -- no leaks are possible
> ==307088== 
> ==307088== For lists of detected and suppressed errors, rerun with: -s
> ==307088== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0

And again no crash with:
$ nix build .#bash5-without-bash-malloc
$ valgrind result/bin/bash --norc -c true
> ==307652== Memcheck, a memory error detector
> ==307652== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
> ==307652== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
> ==307652== Command: result/bin/bash --norc -c true
> ==307652== 
> ==307652== 
> ==307652== HEAP SUMMARY:
> ==307652==     in use at exit: 65,306 bytes in 641 blocks
> ==307652==   total heap usage: 3,078 allocs, 2,437 frees, 223,881 bytes 
> allocated
> ==307652== 
> ==307652== LEAK SUMMARY:
> ==307652==    definitely lost: 0 bytes in 0 blocks
> ==307652==    indirectly lost: 0 bytes in 0 blocks
> ==307652==      possibly lost: 0 bytes in 0 blocks
> ==307652==    still reachable: 65,306 bytes in 641 blocks
> ==307652==         suppressed: 0 bytes in 0 blocks
> ==307652== Rerun with --leak-check=full to see details of leaked memory
> ==307652== 
> ==307652== For lists of detected and suppressed errors, rerun with: -s
> ==307652== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)