Re: String replacements leak small amounts of memory each time

[Matthew Woehlke]

oyvindh@dhampir.no wrote:
> Description:
> When used in a script that iterates over several thousand lines of
> logs or similar data, the bash string replacement functions seem to
> leak memory. The Repeat-By list uses "ls -lR" to generate input, but
> any data will do (try your system logs)
>
> Repeat-By:
>         Start a shell, and start "top" or some other resource monitoring tool
>
>         Try one of the following:
>         while read line; do test=${line%%/*}; done<  <(ls -lR)
>         while read line; do test=${line//a/b}; done<  <(ls -lR)
>         while read line; do test=${line#\ }; done<  <(ls -lR)

No question something bad is going on here.

Using bash 'bash-4.0.35-3.fc12.x86_64' (from Fedora 12); I know it's not 
latest but it's what I have convenient, and anyway the OP is reporting 
the issue against 4.1.5, so it probably hasn't been fixed.

I started two shells (each has exactly processed my startup scripts and 
nothing else); both start out at 109m VIRT.

I then ran:
shell1$ while read line; do temp=$line; done < <(seq 1 5000000)
shell2$ while read line; do temp=${line//1/2}; done < <(seq 1 5000000)

(This was so one could be sure the problem is the replacement and not 
something related to e.g. buffering the command output.)

After both completed, here's what top says:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30042 mwoehlke  20   0  109m 2040 1536 S  0.0  0.1   1:44.21 bash
30097 mwoehlke  20   0  264m 157m 1536 S  0.0  4.0   2:42.62 bash

Notice that not only did shell2 balloon significantly in VIRT, but it 
ate up 157m RES as well, while shell1's RES is practically non-existent. 
(I am confident the RES would drop significantly in the face of memory 
pressure, but it seems to indicate solidly that shell2 touched a much 
larger chunk of memory than shell1.)

Also, when run under valgrind, I see a number of leaks with block count 
exactly equal to the number of lines processed.

The line numbers here are probably off, but FWIW:

==30566== 16 bytes in 1 blocks are definitely lost in loss record 110 of 245
==30566==    at 0x4A0515D: malloc (vg_replace_malloc.c:195)
==30566==    by 0x463F1A: xmalloc (xmalloc.c:86)
==30566==    by 0x428B39: alloc_word_desc (make_cmd.c:92)
==30566==    by 0x428C25: make_bare_word (make_cmd.c:104)
==30566==    by 0x43752B: copy_word (copy_cmd.c:61)
==30566==    by 0x4376F0: copy_word_list (copy_cmd.c:75)
==30566==    by 0x44867F: expand_word_list_internal (subst.c:8736)
==30566==    by 0x460177: redirection_expand (redir.c:253)
==30566==    by 0x460B5F: do_redirection_internal (redir.c:751)
==30566==    by 0x4615C9: do_redirections (redir.c:200)
==30566==    by 0x42EA06: execute_command_internal (execute_cmd.c:678)
==30566==    by 0x468D89: parse_and_execute (evalstring.c:315)
==30566==
==30566== 16 bytes in 1 blocks are definitely lost in loss record 111 of 245
==30566==    at 0x4A0515D: malloc (vg_replace_malloc.c:195)
==30566==    by 0x463F1A: xmalloc (xmalloc.c:86)
==30566==    by 0x428B39: alloc_word_desc (make_cmd.c:92)
==30566==    by 0x428C25: make_bare_word (make_cmd.c:104)
==30566==    by 0x447A96: expand_word_internal (subst.c:8018)
==30566==    by 0x4487CE: expand_word_list_internal (subst.c:8664)
==30566==    by 0x460177: redirection_expand (redir.c:253)
==30566==    by 0x460B5F: do_redirection_internal (redir.c:751)
==30566==    by 0x4615C9: do_redirections (redir.c:200)
==30566==    by 0x42EA06: execute_command_internal (execute_cmd.c:678)
==30566==    by 0x468D89: parse_and_execute (evalstring.c:315)
==30566==    by 0x41B6D9: run_one_command (shell.c:1312)
==30566==
==30566== 79,968 bytes in 4,998 blocks are definitely lost in loss 
record 245 of 245
==30566==    at 0x4A0515D: malloc (vg_replace_malloc.c:195)
==30566==    by 0x463F1A: xmalloc (xmalloc.c:86)
==30566==    by 0x428B39: alloc_word_desc (make_cmd.c:92)
==30566==    by 0x428C25: make_bare_word (make_cmd.c:104)
==30566==    by 0x447A96: expand_word_internal (subst.c:8018)
==30566==    by 0x4487CE: expand_word_list_internal (subst.c:8664)
==30566==    by 0x42DB8E: execute_simple_command (execute_cmd.c:3577)
==30566==    by 0x42EF56: execute_command_internal (execute_cmd.c:738)
==30566==    by 0x42F99D: execute_command (execute_cmd.c:373)
==30566==    by 0x4305DC: execute_while_or_until (execute_cmd.c:3016)
==30566==    by 0x42F0B3: execute_command_internal (execute_cmd.c:2984)
==30566==    by 0x468D89: parse_and_execute (evalstring.c:315)

--
Matthew
Please do not quote my e-mail address unobfuscated in message bodies.
--
What, no punchline?