Re: a very slow problem of replacing string

[Greg Wooledge]

On Thu, Sep 23, 2010 at 10:12:28PM +0900, sky wrote:
> #
> # prepare 1000 strings of 6 digits
> #
> TEST_LIST=`seq 100100 100 200000`
> echo $TEST_LIST | wc

Actually, this is one gigantic string, not 1000 strings.

> #
> # delete "150000"
> #
> T0=$SECONDS
> A=${TEST_LIST//150000}
> T1=$SECONDS
> B=`echo $TEST_LIST | sed s/150000//g`
> T2=$SECONDS

Yes, it's known that operations on very large strings in bash can take
a long time.  (Chet may be able to address that problem; I can't.)

I wonder what you're actually trying to do.  Based on the sample, it
seems that you are trying to remove one element from a set.  If you
want an efficient way to do that, you might consider associative arrays:

declare -A myset
for ((i=100100; i<=200000; i+=100)); do myset[i]=1; done

Now we have an associative array that indicates, for each number,
whether that number is "in" the "set" or not.  (Normal non-associative
arrays could also be used as long as the elements are positive integers,
as they are in this example.  But the associative array can also be used
with other strings as elements.)

Now, if we want to delete the element 150000 from the "set", we simply
do this:

unset 'myset[150000]'

I believe that will be a bit faster than the A=${TEST_LIST//150000} example.