[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Memory leak when catting(/sedding/...) large binary files with backt

From: Ben Taylor
Subject: Re: Memory leak when catting(/sedding/...) large binary files with backticks
Date: Mon, 12 May 2008 10:14:42 +0100
User-agent: Thunderbird (X11/20080226)

Chet Ramey wrote:
address@hidden wrote:
Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-redhat-linux-gnu' -DCONF_VENDOR='redhat' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL -DHAVE_CONFIG_H -I. -I. -I./include -I./lib -D_GNU_SOURCE -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic uname output: Linux pmpc983.npm.ac.uk #1 SMP Sat Mar 29 09:15:49 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-redhat-linux-gnu

Bash Version: 3.2
Patch Level: 33
Release Status: release

Using echo `cat ...` on a large binary file causes lots of memory to be used (fine), but if you ctrl-c while it's running it doesn't die properly and doesn't return used memory when finished. Originally found by screwing up a sed command (can also reproduce bug using sed rather than cat) while trying to rename a group of files.

    Every time
    1. Find large binary data file for test (mine is ~3.2GB)
2. echo `cat filename` 3. Ctrl-C previous command while running (doesn't terminate)
    4. When step 2 eventually returns it does not release memory

I'm not sure what you mean by `doesn't return used memory', but if you mean
a process's size as reported by ps or similar, that does not indicate a
memory leak. A memory leak is memory that has been allocated by a program
to which it retains no handles.

malloc acts as a cache between an application and the kernel.  Memory
obtained from the kernel using malloc may, under some circumstances, be
returned to the kernel upon free, but this may not always be possible.
Memory that is not returned to the kernel by freeing pages or using sbrk
with a negative argument is retained and used to satisfy future requests.

I ran your test using valgrind to check for memory leaks (but with only
a 330 MB file), and it reported no leaks after ^C.


I was just going on what ps reported, but I assumed it was leaking on the basis that the memory did not report as "free" until I kill -9'd the relevant bash process (just kill didn't work). Once it'd been done a couple of times so most of the memory was consumed, it definitely had an adverse effect on performance - even other simple bash commands took several seconds to return a result, which I assume was because they were fighting for memory. The affected bash also didn't show any sub-processes using ps -auxf (shouldn't it have shown a cat process if it was still holding resources?).

If you guys on here reckon it's not a bug that's fine - I admit I'm not exactly au fait with the inner workings of bash, maybe it's working as intended. I just figured since it was eating my memory and not making that memory available to other programs when it was ^C'd (as you would do when you realised you'd inadvertently catted or sedded a 3GB binary file) that I'd report it.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]