Re: Severe memleak in sequence expressions?

[Marc Schiffbauer]

* Bob Proulx schrieb am 01.12.11 um 05:34 Uhr:
> Marc Schiffbauer wrote:
> > Greg Wooledge schrieb:
> > > Marc Schiffbauer wrote:
> > > > echo {0..10000000}>/dev/null
> > > > 
> > > > This makes my system starting to swap as bash will use several GiB of
> > > > memory.
> > >
> > > In my opinion, no.  You're asking bash to generate a list of words from 0
> > > to 1000000000000000000 all at once.  It faithfully attempts to do so.
> > 
> > Yeah, ok but it will not free the mem it allocated later on (see
> > other mail)
> 

Hi Bob,

[...]
> In total to generate all of the arguments for {0..10000000} consumes
> at least 78,888,899 bytes or 75 megabytes of memory(!) if I did all of
> the math right.  Each order of magnitude added grows the amount of
> required memory by an *order of magnitude*.  This should not in any
> way be surprising.  In order to generate 1000000000000000000 arguments
> it might consume 7.8e7 * 1e10 equals 7.8e17 bytes ignoring the smaller
> second order effects.  That is a lot of petabytes of memory!  And it
> is terribly inefficient.  You would never really want to do it this
> way.  You wouldn't want to burn that much memory all at once.  Instead
> you would want to make a for-loop to iterate over the sequence such as
> the "for ((i=1; i<=1000000000000000000; i++)); do" construct that Greg
> suggested.  That is a much more efficient way to do a loop over that
> many items.  And it will execute much faster.  Although a loop of that
> large will take a long time to complete.
> 

I was hit by that by accident. Normally I always do normal for-loops
instead so I was a bit surprised by the fact that my machine was not
responding anymore ;-)

Clearly, when I think about it again, it is more or less obvious.

> Put yourself in a shell author's position.  What would you think of
> this situation?  Trying to generate an unreasonably large number of
> program arguments is, well, unreasonable.  I think this is clearly an
> abuse of the feature.  You can't expect any program to be able to
> generate and use that much memory.

ACK

> And as for whether a program should return unused memory back to the
> operating system for better or worse very few programs actually do it.
> It isn't simple.  It requires more accounting to keep track of memory
> in order to know what can be returned.  It adds to the complexity of
> the code and complexity tends to create bugs.  I would rather have a
> simple and bug free program than one that is full of features but also
> full of bugs.  Especially the shell where bugs are really bad.
> Especially in a case like this where that large memory footprint was
> only due to the unreasonably large argument list it was asked to
> create.  Using a more efficient language construct avoids the memory
> growth, which is undesirable no matter what, and once that memmory
> growth is avoided then there isn't a need to return the memory it
> isn't using to the system either.
> 
> If you want bash to be reduced to a smaller size try exec'ing itself
> in order to do this.
> 
>   $ exec bash
> 
> That is my 2 cents worth plus a little more for free. :-)


Thank you for the explanation. 

I will not consider this a bug anymore ;-)

-Marc
-- 
8AAC 5F46 83B4 DB70 8317  3723 296C 6CCA 35A6 4134