Re: Bash 5 increase in RAM from loop with many interations that append to file

[Chet Ramey]

On 10/27/20 12:11 AM, Scott Kostyshak wrote:
> The following example uses more peak RAM on new bash versions than old 
> versions:
> 
> for i in {1..1000000}; do
>   echo "${i}" >> example.txt
> done
> 
> By measuring peak memory usage with time (/usr/bin/time -f "%E %P %M"),
> I get that newer versions of Bash use about 284M, where older versions
> use about 191M.
> 
> Is this perceived increase in memory usage worth looking more into or is
> it intended?

My guess is that the huge list that results from the brace expansion caused
the bash malloc to cross over into a larger memory bin because of unrelated
memory allocation patterns that changed between bash versions. Once you
allocate that much memory, a binary bin allocation method is going to waste
some. In bash-5.0, that large allocation is going to use mmap, which adds
some overhead of its own.

If you can use valgrind or a similar tool to identify a memory leak, that
would be useful.

Chet
-- 
``The lyf so short, the craft so long to lerne.'' - Chaucer
                 ``Ars longa, vita brevis'' - Hippocrates
Chet Ramey, UTech, CWRU    chet@case.edu    http://tiswww.cwru.edu/~chet/