[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: using mapfile is extreamly slow compared to oldfashinod ways to read
From: |
Lennart Schultz |
Subject: |
Re: using mapfile is extreamly slow compared to oldfashinod ways to read files |
Date: |
Sat, 28 Mar 2009 09:19:08 +0100 |
It seems that mapfile is OK for small numbers but for bigger numbers it
starts to compsume time.
I made a litle test:
rm Xyz; unset MAPFILE # clear
max=9999 # set limit
time for i in $(seq 0 $max); do echo 'Xyz' >> Xyz; done
real 0m0.490s
user 0m0.304s
sys 0m0.124s
time mapfile < Xyz
real 0m0.005s
user 0m0.008s
sys 0m0.000s
time while read line; do echo $line > /dev/null; done < Xyz
real 0m1.124s
user 0m0.456s
sys 0m0.108s
time for i in $(seq 0 $max); do echo echo ${MAPFILE[$i]}> /dev/null; done
real 0m2.184s
user 0m0.976s
sys 0m0.104s
rm Xyz ;unset MAPFILE
max=99999
time for i in $(seq 0 $max); do echo 'Xyz' >> Xyz; done
real 0m8.204s
user 0m3.264s
sys 0m1.188s
time mapfile < Xyz
real 0m0.062s
user 0m0.044s
sys 0m0.000s
time while read line; do echo $line > /dev/null; done < Xyz
real 0m11.328s
user 0m4.500s
sys 0m1.140s
time for i in $(seq 0 $max); do echo echo ${MAPFILE[$i]}> /dev/null; done
real 9m52.832s
user 5m38.305s
sys 0m3.636s
At the time of testing I had sufficient of free memory no swapping, and no
othe time compsuming programs.
2009/3/28 Chris F.A. Johnson <cfaj@freeshell.org>
> On Fri, 27 Mar 2009, Lennart Schultz wrote:
>
> Chris,
>> I agree with you to use the right tool at the right time, and mapfile
>> seems
>> not to be the right tool for my problem, but I will just give you some
>> facts
>> of my observations:
>>
>> using a fast tool like egrep just to find a simple string in my datafile
>> gives the following times:
>>
>> time egrep '<pro' >/dev/null < dr.xml
>>
>> real 0m54.628s
>> user 0m27.310s
>> sys 0m0.036s
>>
>> My original bash script :
>>
>> time xml2e2-loadepg
>>
>> real 1m53.264s
>> user 1m22.145s
>> sys 0m30.674s
>>
>> While the questions seems to go on spawning subshells and the cost I have
>> checked my script
>> it is only calling one external command is date which in total is called a
>> little less than 20000 times. I have just for this test changed the call
>> of
>> date to an assignment of an constant. and now it looks:
>>
>> time xml2e2-loadepg
>>
>> real 1m3.826s
>> user 1m2.700s
>> sys 0m1.004s
>>
>> I also made the same change to the version of the program using mapfile,
>> and
>> changed line=$(echo $i) to
>> line=${i##+([[:space:]])}
>> so the mainloop is absolulty without any sub shell spawns:
>>
>> time xml2e2-loadepg.new
>>
>> real 65m2.378s
>> user 63m16.717s
>> sys 0m1.124s
>>
>
> How much of that is taken by mapfile? Time the mapfile command and
> the loop separately:
>
> time mapfile < file
> time for i in "${MAPFILE[@]}"
>
> --
> Chris F.A. Johnson, webmaster <http://woodbine-gerrard.com>
> ===================================================================
>
> Author:
> Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
>
- using mapfile is extreamly slow compared to oldfashinod ways to read files, Lennart Schultz, 2009/03/26
- Re: using mapfile is extreamly slow compared to oldfashinod ways to read files, Greg Wooledge, 2009/03/26
- Re: using mapfile is extreamly slow compared to oldfashinod ways to read files, Chet Ramey, 2009/03/26
- Re: using mapfile is extreamly slow compared to oldfashinod ways to read files, Chris F.A. Johnson, 2009/03/26
- Re: using mapfile is extreamly slow compared to oldfashinod ways to read files, Greg Wooledge, 2009/03/27
- Re: using mapfile is extreamly slow compared to oldfashinod ways to read files, Lennart Schultz, 2009/03/27