bug-bash
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: **/ directory wildcards leak file descriptors and memory


From: Serge van den Boom
Subject: Re: **/ directory wildcards leak file descriptors and memory
Date: Sat, 23 Jun 2012 23:28:01 +0200 (CEST)
User-agent: Alpine 2.00 (BSF 1167 2008-08-23)

On Sat, 23 Jun 2012, Chet Ramey wrote:
> On 6/20/12 2:50 PM, Serge van den Boom wrote:
> > Bash Version: 4.2
> > Patch Level: 29
> > Release Status: release
> > 
> > Description:
> >     If you interrupt the evaluation of **/ wildcard expansion, the file
> >     descriptors which were opened will not be closed.
> >     Also, memory used will not be released.
> 
> I can't reproduce this:
> 
[...]
> 
> Bash consuming memory is not evidence of a memory leak.  The bash malloc
> only returns memory to the kernel under certain circumstances; it usually
> acts as a cache between the application (bash) and the kernel.  You need
> to run this using a leak-checking tool like valgrind to determine whether
> bash has allocated memory to which it no longer retains a handle.

It may be superfluous now, after Andreas' response, but here is some
additional information.

If you leave the wildcard expansion running for longer, the effect is
greater, and out-of-memory errors will be shown when executing a command
afterwards:

$ echo $BASH_VERSION
4.2.29(1)-release

$ echo $$
16346

$ ls /**/*.txt
[after 15 minutes]
^C

$ ls
bash: fork: Cannot allocate memory
bash: fork: Cannot allocate memory
bash: fork: Cannot allocate memory
[... about 300 more of these ...]
bash: fork: Cannot allocate memory
bash: fork: Cannot allocate memory
bash: fork: Cannot allocate memory
bash: cannot make pipe for command substitution: Too many open files

[ In a different shell instance (one which is still usable): ]
$ ls -fl /proc/16346/fd
total 0K
dr-x------ 2 svdb users  0 Jun 23 22:44 ./
dr-x------ 6 svdb users  0 Jun 23 22:44 ../
lrwx------ 1 svdb users 64 Jun 23 23:06 0 -> /dev/pts/1
lrwx------ 1 svdb users 64 Jun 23 23:06 1 -> /dev/pts/1
lrwx------ 1 svdb users 64 Jun 23 23:06 2 -> /dev/pts/1
l-wx------ 1 svdb users 64 Jun 23 23:06 3 -> /mnt/hda7/home/svdb/.fluxbox/log
lr-x------ 1 svdb users 64 Jun 23 23:06 4 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 5 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 6 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 7 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 8 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 9 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 10 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 11 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 12 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 13 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 14 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 15 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 16 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 17 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 18 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 19 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 20 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 21 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 22 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 23 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 24 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 25 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 26 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 27 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 28 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 29 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 30 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 31 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 32 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 33 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 34 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 35 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 36 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 37 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 38 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 39 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 40 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 41 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 42 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 43 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 44 -> /mnt/
lr-x------ 1 svdb users 64 Jun 23 23:06 45 -> /mnt/hda7/
lr-x------ 1 svdb users 64 Jun 23 23:06 46 -> /mnt/hda7/home/
lr-x------ 1 svdb users 64 Jun 23 23:06 47 -> /mnt/hda7/home/svdb/
lr-x------ 1 svdb users 64 Jun 23 23:06 48 -> /mnt/hda7/home/svdb/c/
lr-x------ 1 svdb users 64 Jun 23 23:06 49 -> /mnt/hda7/home/svdb/c/tests/
lr-x------ 1 svdb users 64 Jun 23 23:06 50 -> /mnt/hda7/home/svdb/c/tests/tar/
lr-x------ 1 svdb users 64 Jun 23 23:06 51 -> 
/mnt/hda7/home/svdb/c/tests/tar/work/
lr-x------ 1 svdb users 64 Jun 23 23:06 52 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 53 -> /mnt/
lr-x------ 1 svdb users 64 Jun 23 23:06 54 -> /mnt/hda7/
lr-x------ 1 svdb users 64 Jun 23 23:06 55 -> /mnt/hda7/home/
lr-x------ 1 svdb users 64 Jun 23 23:06 56 -> /mnt/hda7/home/svdb/
lr-x------ 1 svdb users 64 Jun 23 23:06 57 -> /mnt/hda7/home/svdb/c/
lr-x------ 1 svdb users 64 Jun 23 23:06 58 -> /mnt/hda7/home/svdb/c/cheater/
lr-x------ 1 svdb users 64 Jun 23 23:06 59 -> 
/mnt/hda7/home/svdb/c/cheater/arch/
lr-x------ 1 svdb users 64 Jun 23 23:06 60 -> pipe:[1031416]
l-wx------ 1 svdb users 64 Jun 23 23:06 61 -> pipe:[1031416]
lr-x------ 1 svdb users 64 Jun 23 23:06 62 -> pipe:[1031417]
l-wx------ 1 svdb users 64 Jun 23 23:06 63 -> pipe:[1031417]
lr-x------ 1 svdb users 64 Jun 23 23:06 64 -> pipe:[1031418]
l-wx------ 1 svdb users 64 Jun 23 23:06 65 -> pipe:[1031418]
[...]
lr-x------ 1 svdb users 64 Jun 23 23:06 1019 -> pipe:[1031895]
l-wx------ 1 svdb users 64 Jun 23 23:06 1020 -> pipe:[1031895]
lr-x------ 1 svdb users 64 Jun 23 23:06 1021 -> pipe:[1031896]
l-wx------ 1 svdb users 64 Jun 23 23:06 1022 -> pipe:[1031896]

The file descriptors up to 59 were there after the path name expansion
was interrupted, but I now see that the file descriptors for the pipes
only appeared after I executed 'ls', which might suggest that there is a
second issue.

$ htop
[...]
  PID  PPID USER   PRI  NI  VIRT   RES   SHR S CPU CPU% MEM%   TIME+  Command
16346 16345 svdb    20   0  422M  417M  1488 S   2  0.0 13.8  2:19.19 bash
[...]

Note the 'VIRT' and 'RES' columns.
For a fresh bash, htop shows something like this:
[...]
  PID  PPID USER   PRI  NI  VIRT   RES   SHR S CPU CPU% MEM%   TIME+  Command
17484 17483 svdb    20   0  5404  1736  1312 S   2  0.0  0.1  0:00.02 bash
[...]


Btw, I should add that I compiled with --without-bash-malloc, following
the instructions from
http://www.linuxfromscratch.org/lfs/view/stable/chapter05/bash.html .
I mistakenly thought that my options to 'configure' were included in the
bashbug output. My full 'configure' command line was:
        ./configure --prefix=/opt/bash-4.2 --disable-rpath 
--with-installed-readline=/opt/readline --with-curses --without-bash-malloc
And my readline is version 6.2, with both available patches applied.

Regards,

Serge




reply via email to

[Prev in Thread] Current Thread [Next in Thread]