[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: **/ directory wildcards leak file descriptors and memory
From: |
Serge van den Boom |
Subject: |
Re: **/ directory wildcards leak file descriptors and memory |
Date: |
Sat, 23 Jun 2012 23:28:01 +0200 (CEST) |
User-agent: |
Alpine 2.00 (BSF 1167 2008-08-23) |
On Sat, 23 Jun 2012, Chet Ramey wrote:
> On 6/20/12 2:50 PM, Serge van den Boom wrote:
> > Bash Version: 4.2
> > Patch Level: 29
> > Release Status: release
> >
> > Description:
> > If you interrupt the evaluation of **/ wildcard expansion, the file
> > descriptors which were opened will not be closed.
> > Also, memory used will not be released.
>
> I can't reproduce this:
>
[...]
>
> Bash consuming memory is not evidence of a memory leak. The bash malloc
> only returns memory to the kernel under certain circumstances; it usually
> acts as a cache between the application (bash) and the kernel. You need
> to run this using a leak-checking tool like valgrind to determine whether
> bash has allocated memory to which it no longer retains a handle.
It may be superfluous now, after Andreas' response, but here is some
additional information.
If you leave the wildcard expansion running for longer, the effect is
greater, and out-of-memory errors will be shown when executing a command
afterwards:
$ echo $BASH_VERSION
4.2.29(1)-release
$ echo $$
16346
$ ls /**/*.txt
[after 15 minutes]
^C
$ ls
bash: fork: Cannot allocate memory
bash: fork: Cannot allocate memory
bash: fork: Cannot allocate memory
[... about 300 more of these ...]
bash: fork: Cannot allocate memory
bash: fork: Cannot allocate memory
bash: fork: Cannot allocate memory
bash: cannot make pipe for command substitution: Too many open files
[ In a different shell instance (one which is still usable): ]
$ ls -fl /proc/16346/fd
total 0K
dr-x------ 2 svdb users 0 Jun 23 22:44 ./
dr-x------ 6 svdb users 0 Jun 23 22:44 ../
lrwx------ 1 svdb users 64 Jun 23 23:06 0 -> /dev/pts/1
lrwx------ 1 svdb users 64 Jun 23 23:06 1 -> /dev/pts/1
lrwx------ 1 svdb users 64 Jun 23 23:06 2 -> /dev/pts/1
l-wx------ 1 svdb users 64 Jun 23 23:06 3 -> /mnt/hda7/home/svdb/.fluxbox/log
lr-x------ 1 svdb users 64 Jun 23 23:06 4 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 5 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 6 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 7 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 8 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 9 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 10 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 11 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 12 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 13 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 14 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 15 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 16 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 17 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 18 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 19 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 20 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 21 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 22 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 23 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 24 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 25 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 26 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 27 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 28 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 29 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 30 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 31 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 32 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 33 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 34 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 35 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 36 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 37 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 38 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 39 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 40 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 41 -> /dev/
lr-x------ 1 svdb users 64 Jun 23 23:06 42 -> /proc/16346/fd/
lr-x------ 1 svdb users 64 Jun 23 23:06 43 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 44 -> /mnt/
lr-x------ 1 svdb users 64 Jun 23 23:06 45 -> /mnt/hda7/
lr-x------ 1 svdb users 64 Jun 23 23:06 46 -> /mnt/hda7/home/
lr-x------ 1 svdb users 64 Jun 23 23:06 47 -> /mnt/hda7/home/svdb/
lr-x------ 1 svdb users 64 Jun 23 23:06 48 -> /mnt/hda7/home/svdb/c/
lr-x------ 1 svdb users 64 Jun 23 23:06 49 -> /mnt/hda7/home/svdb/c/tests/
lr-x------ 1 svdb users 64 Jun 23 23:06 50 -> /mnt/hda7/home/svdb/c/tests/tar/
lr-x------ 1 svdb users 64 Jun 23 23:06 51 ->
/mnt/hda7/home/svdb/c/tests/tar/work/
lr-x------ 1 svdb users 64 Jun 23 23:06 52 -> //
lr-x------ 1 svdb users 64 Jun 23 23:06 53 -> /mnt/
lr-x------ 1 svdb users 64 Jun 23 23:06 54 -> /mnt/hda7/
lr-x------ 1 svdb users 64 Jun 23 23:06 55 -> /mnt/hda7/home/
lr-x------ 1 svdb users 64 Jun 23 23:06 56 -> /mnt/hda7/home/svdb/
lr-x------ 1 svdb users 64 Jun 23 23:06 57 -> /mnt/hda7/home/svdb/c/
lr-x------ 1 svdb users 64 Jun 23 23:06 58 -> /mnt/hda7/home/svdb/c/cheater/
lr-x------ 1 svdb users 64 Jun 23 23:06 59 ->
/mnt/hda7/home/svdb/c/cheater/arch/
lr-x------ 1 svdb users 64 Jun 23 23:06 60 -> pipe:[1031416]
l-wx------ 1 svdb users 64 Jun 23 23:06 61 -> pipe:[1031416]
lr-x------ 1 svdb users 64 Jun 23 23:06 62 -> pipe:[1031417]
l-wx------ 1 svdb users 64 Jun 23 23:06 63 -> pipe:[1031417]
lr-x------ 1 svdb users 64 Jun 23 23:06 64 -> pipe:[1031418]
l-wx------ 1 svdb users 64 Jun 23 23:06 65 -> pipe:[1031418]
[...]
lr-x------ 1 svdb users 64 Jun 23 23:06 1019 -> pipe:[1031895]
l-wx------ 1 svdb users 64 Jun 23 23:06 1020 -> pipe:[1031895]
lr-x------ 1 svdb users 64 Jun 23 23:06 1021 -> pipe:[1031896]
l-wx------ 1 svdb users 64 Jun 23 23:06 1022 -> pipe:[1031896]
The file descriptors up to 59 were there after the path name expansion
was interrupted, but I now see that the file descriptors for the pipes
only appeared after I executed 'ls', which might suggest that there is a
second issue.
$ htop
[...]
PID PPID USER PRI NI VIRT RES SHR S CPU CPU% MEM% TIME+ Command
16346 16345 svdb 20 0 422M 417M 1488 S 2 0.0 13.8 2:19.19 bash
[...]
Note the 'VIRT' and 'RES' columns.
For a fresh bash, htop shows something like this:
[...]
PID PPID USER PRI NI VIRT RES SHR S CPU CPU% MEM% TIME+ Command
17484 17483 svdb 20 0 5404 1736 1312 S 2 0.0 0.1 0:00.02 bash
[...]
Btw, I should add that I compiled with --without-bash-malloc, following
the instructions from
http://www.linuxfromscratch.org/lfs/view/stable/chapter05/bash.html .
I mistakenly thought that my options to 'configure' were included in the
bashbug output. My full 'configure' command line was:
./configure --prefix=/opt/bash-4.2 --disable-rpath
--with-installed-readline=/opt/readline --with-curses --without-bash-malloc
And my readline is version 6.2, with both available patches applied.
Regards,
Serge