[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Memory leak when catting(/sedding/...) large binary files with backt
Re: Memory leak when catting(/sedding/...) large binary files with backticks
Mon, 12 May 2008 10:14:42 +0100
Thunderbird 184.108.40.206 (X11/20080226)
Chet Ramey wrote:
I was just going on what ps reported, but I assumed it was leaking on
the basis that the memory did not report as "free" until I kill -9'd the
relevant bash process (just kill didn't work). Once it'd been done a
couple of times so most of the memory was consumed, it definitely had an
adverse effect on performance - even other simple bash commands took
several seconds to return a result, which I assume was because they were
fighting for memory. The affected bash also didn't show any
sub-processes using ps -auxf (shouldn't it have shown a cat process if
it was still holding resources?).
Configuration Information [Automatically generated, do not change]:
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./include
-I./lib -D_GNU_SOURCE -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2
-fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64
uname output: Linux pmpc983.npm.ac.uk 220.127.116.11-64.fc8 #1 SMP Sat Mar
29 09:15:49 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-redhat-linux-gnu
Bash Version: 3.2
Patch Level: 33
Release Status: release
Using echo `cat ...` on a large binary file causes lots of memory
to be used (fine), but if you ctrl-c while it's running
it doesn't die properly and doesn't return used memory when
finished. Originally found by screwing up a sed command (can
also reproduce bug using sed rather than cat) while trying to
rename a group of files.
1. Find large binary data file for test (mine is ~3.2GB)
2. echo `cat filename` 3. Ctrl-C previous command while
running (doesn't terminate)
4. When step 2 eventually returns it does not release memory
I'm not sure what you mean by `doesn't return used memory', but if you
a process's size as reported by ps or similar, that does not indicate a
memory leak. A memory leak is memory that has been allocated by a
to which it retains no handles.
malloc acts as a cache between an application and the kernel. Memory
obtained from the kernel using malloc may, under some circumstances, be
returned to the kernel upon free, but this may not always be possible.
Memory that is not returned to the kernel by freeing pages or using sbrk
with a negative argument is retained and used to satisfy future requests.
I ran your test using valgrind to check for memory leaks (but with only
a 330 MB file), and it reported no leaks after ^C.
If you guys on here reckon it's not a bug that's fine - I admit I'm not
exactly au fait with the inner workings of bash, maybe it's working as
intended. I just figured since it was eating my memory and not making
that memory available to other programs when it was ^C'd (as you would
do when you realised you'd inadvertently catted or sedded a 3GB binary
file) that I'd report it.