Re: xmalloc crash

[Eduardo A . Bustamante López]

On Sat, Jan 06, 2018 at 09:13:17PM -0700, Bob Proulx wrote:
> > You can see that the brk() system call actually succeeds.
> 
> If you are running under the Linux kernel in the default configuration
> then memory overcommit is enabled.
> 
>   $ sysctl vm.overcommit_memory
> 
> With overcommit enabled brk(), and malloc(), will always succeed even
> if there isn't really enough memory.  Neither will fork().  It doesn't
> mean there actually is that much memory available.  Later if the
> kernel runs out of memory to implement the needs then it will fail and
> trigger the OOM Out Of Memory Killer to virtually kill -9 processes
> until enough memory is freed up to balance the books.  Processes
> killed by the OOM have no opportunity to clean up or log anything.

This is not related to Linux VM overcommit / OOM killer.


>From what I can see, Bash's internal memory allocator (lib/malloc/malloc.c,
internal_malloc) is actually unable to allocate the ~580 MiB of VM, even if
there's enough free physical memory to back it up.

See:
  
http://git.savannah.gnu.org/cgit/bash.git/tree/lib/malloc/malloc.c?h=devel&id=ce0469bfbe4176802d4ffd1f272010e14bc4ed81#n777

An easy way to compare is to compile bash with:

 ./configure --with-bash-malloc && make

 vs

 ./configure --without-bash-malloc && make

And then run the reported brace expansion on both.


In my case, the Bash internal allocator fails around 580 MiB. Unlike Bash
compiled to use the system's malloc (in my case, glibc malloc), which allocated
a few GiBs of memory before I had to kill the process (system started to swap).


So there's indeed a limitation in Bash's internal allocator which is enabled by
the configure script in some systems. I don't know if there's a reason for
having that artificial limitation in place though.