[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Memory leak when catting(/sedding/...) large binary files with backt
From: |
Chet Ramey |
Subject: |
Re: Memory leak when catting(/sedding/...) large binary files with backticks |
Date: |
Sat, 10 May 2008 22:01:55 -0400 |
User-agent: |
Thunderbird 2.0.0.14 (Macintosh/20080421) |
benj@pml.ac.uk wrote:
Configuration Information [Automatically generated, do not change]:
Machine: x86_64
OS: linux-gnu
Compiler: gcc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64'
-DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-redhat-linux-gnu'
-DCONF_VENDOR='redhat' -DLOCALEDIR='/usr/share/locale' -DPACKAGE='bash' -DSHELL
-DHAVE_CONFIG_H -I. -I. -I./include -I./lib -D_GNU_SOURCE -O2 -g -pipe
-Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
--param=ssp-buffer-size=4 -m64 -mtune=generic
uname output: Linux pmpc983.npm.ac.uk 2.6.24.4-64.fc8 #1 SMP Sat Mar 29
09:15:49 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux
Machine Type: x86_64-redhat-linux-gnu
Bash Version: 3.2
Patch Level: 33
Release Status: release
Description:
Using echo `cat ...` on a large binary file causes lots of memory to be
used (fine), but if you ctrl-c while it's running
it doesn't die properly and doesn't return used memory when finished.
Originally found by screwing up a sed command (can
also reproduce bug using sed rather than cat) while trying to rename a
group of files.
Repeat-By:
Every time
1. Find large binary data file for test (mine is ~3.2GB)
2. echo `cat filename`
3. Ctrl-C previous command while running (doesn't terminate)
4. When step 2 eventually returns it does not release memory
I'm not sure what you mean by `doesn't return used memory', but if you mean
a process's size as reported by ps or similar, that does not indicate a
memory leak. A memory leak is memory that has been allocated by a program
to which it retains no handles.
malloc acts as a cache between an application and the kernel. Memory
obtained from the kernel using malloc may, under some circumstances, be
returned to the kernel upon free, but this may not always be possible.
Memory that is not returned to the kernel by freeing pages or using sbrk
with a negative argument is retained and used to satisfy future requests.
I ran your test using valgrind to check for memory leaks (but with only
a 330 MB file), and it reported no leaks after ^C.
Chet
--
``The lyf so short, the craft so long to lerne.'' - Chaucer
Live Strong. No day but today.
Chet Ramey, ITS, CWRU chet@case.edu http://cnswww.cns.cwru.edu/~chet/