Re: Using systemd-249's libnss_systemd.so.2 triggers a crash in bash-5.1's malloc.c

[Chet Ramey]

On 10/3/21 11:59 PM, Julien Moutinho wrote:
> Bash Version: 5.1
> Patch Level: 8
> Release Status: release
> Architecture: x86_64-linux
> 
> Description:
> 
> bash-5.1 reaches crashing code paths
> when launched by systemd-249 or valgrind.
> I cannot get such crashes when bash is built using:
> ./configure --without-bash-malloc

I suspect this is a buffer overflow introduced between systemd-247 and
systemd-249. It's not caught when building bash without the bash malloc
because the default libc malloc probably doesn't do the bounds checking
the bash malloc does, even without malloc debugging turned on.

It may be, but probably is not, due to the single change between the
bash-5.0 and bash-5.1 malloc: aligning returned memory on 16-byte
boundaries on systems with 64-bit pointers. (There is another change
between bash-4.4 and bash-5.0, using mmap for large allocations, but it's
not a factor here.)

I'll explain my reasoning as we go.

> Here's a Nix recipe for reproducible builds,
> but you should be able to reproduce the crash with your own build method.

I use the devel branch, which already has these defines, but I don't have
any handy systems running systemd-249. Anyway, onward.

>> malloc: unknown:0: assertion botched
>> malloc: 0x37c8fa10: allocated: last allocated from unknown:0
>> realloc: start and end chunk sizes differ
>> last command: (null)
>> Aborting...

This error tells us that realloc failed because the stored sizes of the
allocated block, which are stored before and after the block, don't match.
The bash malloc keeps track of the requested size of the block and uses
it in various checks.

The fact that the last allocation location is `unknown:0' means that
the allocation wasn't called directly by bash.

The important thing looks to be that bash is linked against systemd-249's
version of the nss library, and goes through systemd for the password
library functions. Non-systemd versions of the getpw functions don't
seem to exhibit this problem. systemd-247 doesn't exhibit this problem.

>> #3  0x000000000049e408 in xbotch (mem=mem@entry=0x37c8fa10, e=e@entry=8, 
>>     s=0x4b9708 "realloc: start and end chunk sizes differ", 
>> file=file@entry=0x0, line=line@entry=0) at malloc.c:376
>> #4  0x000000000049f5c3 in internal_realloc (mem=0x37c8fa10, n=218, 
>> file=file@entry=0x0, line=line@entry=0, 
>>     flags=flags@entry=0) at malloc.c:1150
>> #5  0x000000000049f867 in realloc (mem=<optimized out>, nbytes=<optimized 
>> out>) at malloc.c:1440
>> #6  0x0000619871c02526 in greedy_realloc ()
>>    from 
>> /nix/store/5iw38x4l9nyp2srnlxfi9l1q9np0pchq-systemd-249.4/lib/libnss_systemd.so.2
>> #7  0x0000619871c04c44 in read_line_full ()
>>    from 
>> /nix/store/5iw38x4l9nyp2srnlxfi9l1q9np0pchq-systemd-249.4/lib/libnss_systemd.so.2
>> #8  0x0000619871c04eb6 in read_one_line_file ()
>>    from 
>> /nix/store/5iw38x4l9nyp2srnlxfi9l1q9np0pchq-systemd-249.4/lib/libnss_systemd.so.2
>> #9  0x0000619871bfd1b8 in proc_cmdline_parse ()
>>    from 
>> /nix/store/5iw38x4l9nyp2srnlxfi9l1q9np0pchq-systemd-249.4/lib/libnss_systemd.so.2
>> #10 0x0000619871bfa939 in log_parse_environment ()
>>    from 
>> /nix/store/5iw38x4l9nyp2srnlxfi9l1q9np0pchq-systemd-249.4/lib/libnss_systemd.so.2
>> #11 0x0000619871b965a7 in __pthread_once_slow (once_control=0x619871c490a4 
>> <once>, 
>>     init_routine=0x619871bf3b20 <setup_logging>) at pthread_once.c:116
>> #12 0x0000619871bf40ff in _nss_systemd_getpwuid_r ()
>>    from 
>> /nix/store/5iw38x4l9nyp2srnlxfi9l1q9np0pchq-systemd-249.4/lib/libnss_systemd.so.2
>> #13 0x0000619871ee7b46 in __getpwuid_r (uid=uid@entry=65483, 
>> resbuf=resbuf@entry=0x619871fdbf40 <resbuf>, 
>>     buffer=<optimized out>, buflen=buflen@entry=1024, 
>> result=result@entry=0x7b3c23e18658) at ../nss/getXXbyYY_r.c:274
>> #14 0x0000619871ee7378 in getpwuid (uid=65483) at ../nss/getXXbyYY.c:135
>> #15 0x000000000041c9d2 in get_current_user_info () at shell.c:1869
>> #16 0x000000000041cc8a in shell_initialize () at shell.c:1932
>> #17 0x000000000041d1e8 in main (argc=2, argv=0x7b3c23e18918, 
>> env=0x7b3c23e18930) at shell.c:572

This tells us that this is internal to the systemd-249 getpwuid -- the
buffer is allocated there and reallocated there. There's no bash code
executed.

That points the finger at systemd-249 a little more -- bash built with
its internal malloc doesn't have a problem with realloc unless systemd
is running its getpwuid implementation. This is the case when bash is
linked against the functions from systemd-247.

> AFAICS:
> - No crash happens with bash5-with-bash-malloc on systemd-247.

So it doesn't appear to be bash (nor the bash malloc), since the bash
version is the same, the bash malloc is the same, and the code paths
are the same (getpwuid -> systemd nss code). The only thing that's
changed is the systemd version.

> - No crash happens with bash5-without-bash-malloc on systemd-247 or 
> systemd-249.

Because the default malloc doesn't do this checking.

> - A slightly different crash, still involving _nss_systemd_getpwuid_r,
>   happens with bash4-with-bash-malloc on systemd-247.
>   It might have been fixed when a new definition of MALIGN_MASK
>   was introduced in bash-5.1:

This is the change from 8-byte to 16-byte alignment. One consequence of
this change is that the additional space in the block header is used to
catch underflow. Since there's no underflow message, we can assume the
sizes don't match because the overflow overwrote the size information
stored after the block.

>   
> https://git.savannah.gnu.org/cgit/bash.git/commit/lib/malloc/malloc.c?id=8868edaf2250e09c4e9a1c75ffe3274f28f38581
>   See also: https://www.mail-archive.com/bug-bash@gnu.org/msg24306.html

This was one of the things that prompted the change between bash-5.0 and
bash-5.1.

> - No crash happens with bash4-without-bash-malloc on systemd-247 or 
> systemd-249.

For the same reason as above.

> - bash crashes inside valgrind too,
>   but apparently something different is happening
>   because it crashes even without systemd being involved:

This is a red herring.

-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/