[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
bash sucks lots of RAM doing read & echo in loop
From: |
wojtek |
Subject: |
bash sucks lots of RAM doing read & echo in loop |
Date: |
Mon, 4 Feb 2008 20:05:17 +0100 (CET) |
Configuration Information [Automatically generated, do not change]:
Machine: i386
OS: freebsd6.2
Compiler: cc
Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='i386'
-DCONF_OSTYPE='freebsd6.2' -DCONF_MACHTYPE='i386-portbld-freebsd6.2'
-DCONF_VENDOR='portbld' -DLOCALEDIR='/usr/local/share/locale' -DPACKAGE='bash'
-DSHELL -DHAVE_CONFIG_H -I. -I. -I./include -I./lib -I/usr/local/include
-O2 -fno-strict-aliasing -pipe -march=pentium3
uname output: FreeBSD wojtek.tensor.gdynia.pl 6.2-RELEASE-p10 FreeBSD
6.2-RELEASE-p10 #0: Tue Jan 15 16:35:37 CET 2008
root@FreeSBIE.LiveCD:/usr/src/sys/i386/compile/p234 i386
Machine Type: i386-portbld-freebsd6.2
Bash Version: 3.2
Patch Level: 17
Release Status: release
Description:
bash keeps all output it echoed in memory.
in my case it sucked over 3GB VM processing 2.5GB file.
details in "Repeat-By:"
Repeat-By:
such fragment of my script is executed
while read ll;do echo $RANDOM $RANDOM $ll; done <$tmp2 | $SORT | cut
-f 3- -d " " >$tmp3
where $tmp2 is 2.5GB input file, $tmp3 is output file and $SORT is
"sort -S 512m"
this routine has to randomize lines of input file $tmp2.
and it works fine, but WHY bash gets over 3GB size, which grows as it
processes $tmp2
input?
it should not keep anything in memory but pipe to sort program (which
takes 512MB as
i wanted it+tempfiles). but it does both.
Fix:
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- bash sucks lots of RAM doing read & echo in loop,
wojtek <=