bug-bash
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

big 'list' consumes all memory


From: Mart Frauenlob
Subject: big 'list' consumes all memory
Date: Thu, 16 Apr 2009 11:11:10 +0200
User-agent: Thunderbird 2.0.0.21 (Windows/20090302)

Hello,

today while playing around with brace expansion, I ran into something for me unexpected. The actual intension was to compare the speed / system usage of `seq x y' and brace expansion {x..y}. So I took a fairly large integer (lets say 15755500), and ran the following:

seq 0 15755500

- no problem

printf "%s\n" {0..15755500}

(or for i in {0..15755500}; do echo $i; done)

- cpu and mem usage goes up, system starts swaping hardly, takes a while - bang - bash process killed - out of memory.

System is Debian etch.
GNU bash, version 3.1.17(1)-release (i486-pc-linux-gnu)
600MHz Celeron, 392MB RAM, 512MB swap

I start thinking about it:
ok, building a 'list', which is too big for my system, may always lead to that. So I try:

for i in $(seq 0 15755500); do echo $i; done
-bash: xrealloc: ../bash/subst.c:512: cannot reallocate 182209024 bytes (0 bytes allocated)


ok, thesis looks confirmed...
I'm no C programmer, but I try to think logically about it.
There may be no way around it, as it may be necessary to build the complete list, before it's possible to work with it. If this assumption is correct, as building the list needs to be done in memory -> bang comes sooner or later, depending on the system.

Am I thinking right?

Okay, assuming I'm somehow right, taking that subject to my 'real life scripting', how should I deal with that?

There may be circumstances, where a 'list' becomes fairly large, and if the system is for what ever reason low on resources, the process death is predestined.

I haven't tested any other bash versions, nor other shells (would need to install).
But, I just guess, it will be the same there?

How do experienced bash (shell) programmers deal with such cases?

Are there any guidelines (I may have missed reading) for 'careful' scripting, to avoid such problems?

Thank you for any information!

Greets

Mart




reply via email to

[Prev in Thread] Current Thread [Next in Thread]