Re: Memory leak when catting(/sedding/...) large binary files with backticks

[Ben Taylor]

Chet Ramey wrote:
> benj@pml.ac.uk wrote:
> > Configuration Information [Automatically generated, do not change]:
> > Machine: x86_64
> > OS: linux-gnu
> > Compiler: gcc
> > Compilation CFLAGS:  -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' 
> > -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-redhat-linux-gnu' 
> > -DCONF_VENDOR='redhat' -DLOCALEDIR='/usr/share/locale' 
> > -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H   -I.  -I. -I./include 
> > -I./lib  -D_GNU_SOURCE  -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 
> > -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 
> > -mtune=generic
> > uname output: Linux pmpc983.npm.ac.uk 2.6.24.4-64.fc8 #1 SMP Sat Mar 
> > 29 09:15:49 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
> > Machine Type: x86_64-redhat-linux-gnu
> >
> > Bash Version: 3.2
> > Patch Level: 33
> > Release Status: release
> >
> > Description:
> >     Using echo `cat ...` on a large binary file causes lots of memory 
> > to be used (fine), but if you ctrl-c while it's running
> >     it doesn't die properly and doesn't return used memory when 
> > finished. Originally found by screwing up a sed command (can
> >     also reproduce bug using sed rather than cat) while trying to 
> > rename a group of files.
> >
> > Repeat-By:
> >     Every time
> >     1. Find large binary data file for test (mine is ~3.2GB)
> >     2. echo `cat filename`     3. Ctrl-C previous command while 
> > running (doesn't terminate)
> >     4. When step 2 eventually returns it does not release memory
>
> I'm not sure what you mean by `doesn't return used memory', but if you 
> mean
> a process's size as reported by ps or similar, that does not indicate a
> memory leak.  A memory leak is memory that has been allocated by a 
> program
> to which it retains no handles.
>
> malloc acts as a cache between an application and the kernel.  Memory
> obtained from the kernel using malloc may, under some circumstances, be
> returned to the kernel upon free, but this may not always be possible.
> Memory that is not returned to the kernel by freeing pages or using sbrk
> with a negative argument is retained and used to satisfy future requests.
>
> I ran your test using valgrind to check for memory leaks (but with only
> a 330 MB file), and it reported no leaks after ^C.
>
> Chet
I was just going on what ps reported, but I assumed it was leaking on 
the basis that the memory did not report as "free" until I kill -9'd the 
relevant bash process (just kill didn't work). Once it'd been done a 
couple of times so most of the memory was consumed, it definitely had an 
adverse effect on performance - even other simple bash commands took 
several seconds to return a result, which I assume was because they were 
fighting for memory. The affected bash also didn't show any 
sub-processes using ps -auxf (shouldn't it have shown a cat process if 
it was still holding resources?).

If you guys on here reckon it's not a bug that's fine - I admit I'm not 
exactly au fait with the inner workings of bash, maybe it's working as 
intended. I just figured since it was eating my memory and not making 
that memory available to other programs when it was ^C'd (as you would 
do when you realised you'd inadvertently catted or sedded a 3GB binary 
file) that I'd report it.

Ben