Re: Bash 5 increase in RAM from loop with many interations that append to file

[Scott Kostyshak]

On Tue, Oct 27, 2020 at 09:40:53AM -0400, Chet Ramey wrote:
> On 10/27/20 12:11 AM, Scott Kostyshak wrote:
> > The following example uses more peak RAM on new bash versions than old 
> > versions:
> > 
> > for i in {1..1000000}; do
> >   echo "${i}" >> example.txt
> > done
> > 
> > By measuring peak memory usage with time (/usr/bin/time -f "%E %P %M"),
> > I get that newer versions of Bash use about 284M, where older versions
> > use about 191M.
> > 
> > Is this perceived increase in memory usage worth looking more into or is
> > it intended?
> 
> My guess is that the huge list that results from the brace expansion caused
> the bash malloc to cross over into a larger memory bin because of unrelated
> memory allocation patterns that changed between bash versions. Once you
> allocate that much memory, a binary bin allocation method is going to waste
> some. In bash-5.0, that large allocation is going to use mmap, which adds
> some overhead of its own.
> 
> If you can use valgrind or a similar tool to identify a memory leak, that
> would be useful.

Thanks for the explanation, Chet. I would normally jump at the chance to help 
out because I enjoy digging into things like this, but these days I just can't 
find the time. I will come back to this at some point, but possibly not for a 
while. I'm sorry about that.

Thanks again for all of your work on Bash!

Scott