Re: using mapfile is extreamly slow compared to oldfashinod ways to read files

[Chris F.A. Johnson]

On Thu, 26 Mar 2009, Lennart Schultz wrote:

> I have a bash script which reads about 250000 lines of xml code generating
> about 850 files with information extracted from the xml file.
> It uses the construct:
>
> while read line
> do
>   case "$line" in
>   ....
> done < file
>
> and this takes a little less than 2 minutes
>
> Trying to use mapfile I changed the above construct to:
>
> mapfile  < file
> for i in "${MAPFILE[@]}"
> do
>   line=$(echo $i) # strip leading blanks
>   case "$line" in
>   ....
> done
>
> With this change the job now takes more than 48 minutes. :(

   As has already been suggested, the time it almost certainly taken
   up in the command substitution which you perform on every line.

   If you want to remove leading spaces, it would be better to use a
   single command to do that before reading with mapfile, e,g,:

mapfile < <(sed 's/^ *//' file)

   If you want to remove trailing spaces as well:

mapfile < <(sed -e 's/^ *//' -e 's/ *$//' file)

   Chet, how about an option to mapfile that strips leading and/or
   trailing spaces?

   Another useful option would be to remove newlines.

--
   Chris F.A. Johnson, webmaster         <http://woodbine-gerrard.com>
   ========= Do not reply to the From: address; use Reply-To: ========
   Author:
   Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)