Memory leak in subshell + read before bash 4.4

bug-bash

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Memory leak in subshell + read before bash 4.4

From:	Adrien Mahieux
Subject:	Memory leak in subshell + read before bash 4.4
Date:	Fri, 3 Nov 2017 23:01:32 +0100

Hello,

I think I've found a bug in the loop management. Maybe it's an
expected behavior (didn't find any related topic on the manpages) but
it's annoying in long-running scripts.

The leak is triggered by this idiom (wether or not jobcontrol + lastpipe) :
cmd | read var

but not by this one :
read var < <(cmd)

Tested with :
- bash-4.2.46-19.el7.x86_64
- bash-4.3.42-5.fc23.x86_64
- bash-4.3-48.x86_64

>From the version tested, it was fixed in version 4.4 (tested 4.4-beta)
Do you have an idea of the commit that fixed this behavior, so the fix
can be backported to older versions ?



Here's some code to check this behavior:
#!/bin/bash

# use strict
set -u
# Also fails with no monitor + lastpipe
#set +m
#shopt -s lastpipe

typeset -i i=0
typeset -i max=10000

function getrss { grep VmRSS /proc/$$/status; }
function getdata { echo 1 2 3 4  }

typeset rss_start="$(getrss)"
echo "Before loop: RSS = $rss_start"

while [[ $i -lt $max ]]; do

typeset rss_curr="$(getrss)"
[[ "$rss_start" != "$rss_curr" ]] && {
echo "New mem usage at loop $i: $rss_start => $rss_curr"
rss_start=$rss_curr
}

# First half, let's try the working method
if [[ $i -lt $(( $max/2 )) ]]; then
# That bashism is fine
read some data < <(getdata)
else
# That one leaks something
getdata | read some data
fi

i=i+1
done



and here's the output :

$ bash-4.3.42-5.fc23.x86_64 ~/bin/leak.sh
Before loop: RSS = VmRSS:    2704 kB
New mem usage at loop 1: VmRSS:    2704 kB => VmRSS:    2984 kB
New mem usage at loop 2: VmRSS:    2984 kB => VmRSS:    2988 kB
New mem usage at loop 5069: VmRSS:    2988 kB => VmRSS:    2992 kB
New mem usage at loop 5198: VmRSS:    2992 kB => VmRSS:    2996 kB
New mem usage at loop 5318: VmRSS:    2996 kB => VmRSS:    3000 kB
New mem usage at loop 5447: VmRSS:    3000 kB => VmRSS:    3004 kB
New mem usage at loop 5574: VmRSS:    3004 kB => VmRSS:    3008 kB
New mem usage at loop 5703: VmRSS:    3008 kB => VmRSS:    3012 kB
New mem usage at loop 5831: VmRSS:    3012 kB => VmRSS:    3016 kB
New mem usage at loop 5959: VmRSS:    3016 kB => VmRSS:    3020 kB
New mem usage at loop 6087: VmRSS:    3020 kB => VmRSS:    3024 kB
New mem usage at loop 6215: VmRSS:    3024 kB => VmRSS:    3028 kB
New mem usage at loop 6343: VmRSS:    3028 kB => VmRSS:    3032 kB
New mem usage at loop 6471: VmRSS:    3032 kB => VmRSS:    3036 kB
New mem usage at loop 6599: VmRSS:    3036 kB => VmRSS:    3040 kB
New mem usage at loop 6727: VmRSS:    3040 kB => VmRSS:    3044 kB
New mem usage at loop 6855: VmRSS:    3044 kB => VmRSS:    3048 kB
New mem usage at loop 6982: VmRSS:    3048 kB => VmRSS:    3052 kB
New mem usage at loop 7111: VmRSS:    3052 kB => VmRSS:    3056 kB
New mem usage at loop 7246: VmRSS:    3056 kB => VmRSS:    3060 kB
New mem usage at loop 7361: VmRSS:    3060 kB => VmRSS:    3064 kB
New mem usage at loop 7497: VmRSS:    3064 kB => VmRSS:    3068 kB
New mem usage at loop 7627: VmRSS:    3068 kB => VmRSS:    3072 kB
New mem usage at loop 7757: VmRSS:    3072 kB => VmRSS:    3076 kB
New mem usage at loop 7883: VmRSS:    3076 kB => VmRSS:    3080 kB
New mem usage at loop 8010: VmRSS:    3080 kB => VmRSS:    3084 kB
New mem usage at loop 8139: VmRSS:    3084 kB => VmRSS:    3088 kB
New mem usage at loop 8268: VmRSS:    3088 kB => VmRSS:    3092 kB
New mem usage at loop 8393: VmRSS:    3092 kB => VmRSS:    3096 kB
New mem usage at loop 8520: VmRSS:    3096 kB => VmRSS:    3100 kB
New mem usage at loop 8648: VmRSS:    3100 kB => VmRSS:    3104 kB
New mem usage at loop 8776: VmRSS:    3104 kB => VmRSS:    3108 kB
New mem usage at loop 8904: VmRSS:    3108 kB => VmRSS:    3112 kB
New mem usage at loop 9033: VmRSS:    3112 kB => VmRSS:    3116 kB
New mem usage at loop 9161: VmRSS:    3116 kB => VmRSS:    3120 kB
New mem usage at loop 9289: VmRSS:    3120 kB => VmRSS:    3124 kB
New mem usage at loop 9411: VmRSS:    3124 kB => VmRSS:    3128 kB
New mem usage at loop 9539: VmRSS:    3128 kB => VmRSS:    3132 kB
New mem usage at loop 9666: VmRSS:    3132 kB => VmRSS:    3136 kB
New mem usage at loop 9795: VmRSS:    3136 kB => VmRSS:    3140 kB
New mem usage at loop 9923: VmRSS:    3140 kB => VmRSS:    3144 kB


$./bash-4.3.48 ~/bin/leak.sh
Before loop: RSS = VmRSS:    2708 kB
New mem usage at loop 0: VmRSS:    2708 kB => VmRSS:    3004 kB
New mem usage at loop 5007: VmRSS:    3004 kB => VmRSS:    3008 kB
New mem usage at loop 5135: VmRSS:    3008 kB => VmRSS:    3012 kB
New mem usage at loop 5263: VmRSS:    3012 kB => VmRSS:    3016 kB
New mem usage at loop 5391: VmRSS:    3016 kB => VmRSS:    3020 kB
New mem usage at loop 5519: VmRSS:    3020 kB => VmRSS:    3024 kB
New mem usage at loop 5647: VmRSS:    3024 kB => VmRSS:    3028 kB
New mem usage at loop 5775: VmRSS:    3028 kB => VmRSS:    3032 kB
New mem usage at loop 5903: VmRSS:    3032 kB => VmRSS:    3036 kB
New mem usage at loop 6031: VmRSS:    3036 kB => VmRSS:    3040 kB
New mem usage at loop 6159: VmRSS:    3040 kB => VmRSS:    3044 kB
New mem usage at loop 6287: VmRSS:    3044 kB => VmRSS:    3048 kB
New mem usage at loop 6415: VmRSS:    3048 kB => VmRSS:    3052 kB
New mem usage at loop 6543: VmRSS:    3052 kB => VmRSS:    3056 kB
New mem usage at loop 6671: VmRSS:    3056 kB => VmRSS:    3060 kB
New mem usage at loop 6799: VmRSS:    3060 kB => VmRSS:    3064 kB
New mem usage at loop 6927: VmRSS:    3064 kB => VmRSS:    3068 kB
New mem usage at loop 7055: VmRSS:    3068 kB => VmRSS:    3072 kB
New mem usage at loop 7183: VmRSS:    3072 kB => VmRSS:    3076 kB
New mem usage at loop 7311: VmRSS:    3076 kB => VmRSS:    3080 kB
New mem usage at loop 7438: VmRSS:    3080 kB => VmRSS:    3084 kB
New mem usage at loop 7567: VmRSS:    3084 kB => VmRSS:    3088 kB
New mem usage at loop 7695: VmRSS:    3088 kB => VmRSS:    3092 kB
New mem usage at loop 7823: VmRSS:    3092 kB => VmRSS:    3096 kB
New mem usage at loop 7951: VmRSS:    3096 kB => VmRSS:    3100 kB
New mem usage at loop 8079: VmRSS:    3100 kB => VmRSS:    3104 kB
New mem usage at loop 8207: VmRSS:    3104 kB => VmRSS:    3108 kB
New mem usage at loop 8335: VmRSS:    3108 kB => VmRSS:    3112 kB
New mem usage at loop 8463: VmRSS:    3112 kB => VmRSS:    3116 kB
New mem usage at loop 8591: VmRSS:    3116 kB => VmRSS:    3120 kB
New mem usage at loop 8719: VmRSS:    3120 kB => VmRSS:    3124 kB
New mem usage at loop 8847: VmRSS:    3124 kB => VmRSS:    3128 kB
New mem usage at loop 8975: VmRSS:    3128 kB => VmRSS:    3132 kB
New mem usage at loop 9103: VmRSS:    3132 kB => VmRSS:    3136 kB
New mem usage at loop 9231: VmRSS:    3136 kB => VmRSS:    3140 kB
New mem usage at loop 9359: VmRSS:    3140 kB => VmRSS:    3144 kB
New mem usage at loop 9487: VmRSS:    3144 kB => VmRSS:    3148 kB
New mem usage at loop 9615: VmRSS:    3148 kB => VmRSS:    3152 kB
New mem usage at loop 9743: VmRSS:    3152 kB => VmRSS:    3156 kB
New mem usage at loop 9871: VmRSS:    3156 kB => VmRSS:    3160 kB
New mem usage at loop 9999: VmRSS:    3160 kB => VmRSS:    3164 kB

$ ./bash-4.4-beta ~/bin/leak.sh
Before loop: RSS = VmRSS:    3188 kB
New mem usage at loop 1: VmRSS:    3188 kB => VmRSS:    3268 kB
New mem usage at loop 2: VmRSS:    3268 kB => VmRSS:    3308 kB
New mem usage at loop 3: VmRSS:    3308 kB => VmRSS:    3320 kB
New mem usage at loop 4: VmRSS:    3320 kB => VmRSS:    3344 kB
New mem usage at loop 5: VmRSS:    3344 kB => VmRSS:    3368 kB
New mem usage at loop 6: VmRSS:    3368 kB => VmRSS:    3380 kB
New mem usage at loop 7: VmRSS:    3380 kB => VmRSS:    3396 kB
New mem usage at loop 8: VmRSS:    3396 kB => VmRSS:    3416 kB
New mem usage at loop 9: VmRSS:    3416 kB => VmRSS:    3440 kB
New mem usage at loop 10: VmRSS:    3440 kB => VmRSS:    3452 kB
New mem usage at loop 11: VmRSS:    3452 kB => VmRSS:    3472 kB
New mem usage at loop 12: VmRSS:    3472 kB => VmRSS:    3496 kB
New mem usage at loop 13: VmRSS:    3496 kB => VmRSS:    3508 kB
New mem usage at loop 14: VmRSS:    3508 kB => VmRSS:    3524 kB
New mem usage at loop 15: VmRSS:    3524 kB => VmRSS:    3544 kB
New mem usage at loop 16: VmRSS:    3544 kB => VmRSS:    3568 kB
New mem usage at loop 17: VmRSS:    3568 kB => VmRSS:    3580 kB
New mem usage at loop 18: VmRSS:    3580 kB => VmRSS:    3600 kB
New mem usage at loop 19: VmRSS:    3600 kB => VmRSS:    3608 kB
New mem usage at loop 20: VmRSS:    3608 kB => VmRSS:    3640 kB

$ ./bash-4.4 ~/bin/leak.sh
Before loop: RSS = VmRSS:    2744 kB
New mem usage at loop 0: VmRSS:    2744 kB => VmRSS:    3004 kB
New mem usage at loop 1: VmRSS:    3004 kB => VmRSS:    3008 kB
New mem usage at loop 2: VmRSS:    3008 kB => VmRSS:    3012 kB
New mem usage at loop 4: VmRSS:    3012 kB => VmRSS:    3016 kB
New mem usage at loop 5001: VmRSS:    3016 kB => VmRSS:    4056 kB


cheers,
Adrien

[Prev in Thread]

Current Thread

[Next in Thread]

Memory leak in subshell + read before bash 4.4, Adrien Mahieux <=
- Re: Memory leak in subshell + read before bash 4.4, Chet Ramey, 2017/11/03
  - Re: Memory leak in subshell + read before bash 4.4, Adrien Mahieux, 2017/11/03
    - Re: Memory leak in subshell + read before bash 4.4, Chet Ramey, 2017/11/03
    - Re: Memory leak in subshell + read before bash 4.4, Adrien Mahieux, 2017/11/03
    - Re: Memory leak in subshell + read before bash 4.4, Chet Ramey, 2017/11/04

Prev by Date: Uncomplete multiline PS1 after 'display-shell-version'
Next by Date: building: recutils bash builtins/readrec.so; ./configure --with-bash-headers=... # fails
Previous by thread: Uncomplete multiline PS1 after 'display-shell-version'
Next by thread: Re: Memory leak in subshell + read before bash 4.4
Index(es):
- Date
- Thread