Re: fork costs...

[Eric Blake]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Please keep replies on the list.

According to Kyle Sallee on 10/21/2006 1:43 AM:
> I was looking at
> /libltdl/config/ltmain.m4sh
> one of the first uses of tem_deplibs has a nearby call to
> func_stripname '-R' '' "$libdir"
> That uses a separate invocation of sed
> everytime -R is encountered.
> Sort of expensive if there is more than one occurance.
> 
> Why not fix them all at once?
> 
> For example the xrpath string could still be built
> without calling func_stripname.
> Finally,
> 
> if  test  -n "$xrpath";
> then  xrpath="` echo  "$xrpath" | sort -u | sed "s/^-R//"`"
> fi
> 
> Well, my point is that since the func_stripname
> requires an invocation of sed then all the -R
> should be stripped with a single invocation of sed
> rather than on every occurance where $libdir begins with -R
> 
> BTW, I like how func_stripname reutnrs $func_stripname_results
> instead of echoing it and then having to gather it using an
> assignment to the output of a subshell.
> That definitely helps speed it up.
> 
> If dependency_libs is a long list it might still be fast
> to process it in two separate pipes of commands.
> The first can gather the -R
> the second can gather the rest.
> 
> Instead of doing temp_deplibs"$temp_deplibs $libdir" ;;
> on each itteration...
> dependency_libs="` echo $temp_deplibs | sed "s/-R[^ ]*::g" `"
> 
> Counting echo as a process...
> The above requires 3 process.
> Likewise, dependency_libs could be determined 3 processes.
> 3 processes to determine duplicate dependency libs.
> 
> Even if that seems a bit expensive for
> calls of libtool not involving a lot of librarie
> at least the potential invocations of sed
> to remove the -R could be reduced to only 1.
> 
> There are places for matching duplicates such as:
> 
> *" $deplib "*) specialdeplibs="$specialdeplibs $deplib" ;;
>                   *" $temp_xrpath "*) ;;
> 
> That is incredibly expensive to do when the string is long.
> 
> ltmain.m4sh
> 
> is rather complex.
> 
> Perhaps more functions would be helpful?
> 
> What about if the different major tasks of the
> libtool script were put in separate scripts?
> The invocation of libtool could determine which one
> of those other scripts should be called depending upon
> the desired task and the platform the task will run on?
> 
> The complexity currently is daunting.
> I see where it scoops out the -R
> and then a page down the -L and so forth...
> 
> Encoding the methods in BASH using the for
> loops and case statements and so forth are rather
> lengthy and complex compared to a 3 process method
> involving a subshell assignment.
> However, it could be further reduced if
> instead of assigning those to bash variables if
> the output was piped to a temporary file.
> 
> echo $temp_deplibs | sed "s/-R[^ ]*::g" > /tmp/libtool_something.$tmp
> 
> one invocation of echo one invocation of sed.
> The sed statement could be more complex to eliminate both the -R and -L
> or whatever else may be undesirable in creating that list.
> 
> While I prefer one item per line instead of all items on a line
> separated by space
> the all items on a line separated by a space has one advantage.
> I know this works in bash, but I do not know about /bin/sh
> 
> read < file
> 
> The above will put into $REPLY the entire first line of file
> without using a subshell.
> It is faster than...
> REPLY="$(< file )"
> It is faster than
> REPLY="$( cat file )"
> 
> But it only retrieves one line.
> 
> But using multi-lined streams means
> that sort and uniq could become useful.
> Those case statements to identify duplicates is signifigantly slow.
> 
> It seems as if libtool could be modified to both invoke less processes
> when running and also eliminate some of the slow shell syntax.
> 
> I understand that you are strongly against using forks
> where the equivilent can be coded without it.
> But in some of those instances I have examined there
> are multiple invocations of programs such as sed
> which means that there are several fork potentials.
> If no greater change is made then at least those
> instances where an item is modified before it is
> put into a a string of variables it can be made so
> that the string of variables is made and then the
> entire string is modified with a single invocation of sed.
> 
> In that 26M trace I have.
> I counted
> 2432 instances of /bin/sed
> That is a lot of forking.
> 
> What would be helpful is to look for instances of code
> that is within a loop where a fork is performed
> for each instance of a certain case statement.
> Then, build the string without the fork
> and after the string is built if it even exists
> then run sed once to make it the way it should be, yes?
> 
> I would like to know more about how costly forks are
> on the platforms where it is seriously slow, please.
> 2432 invocations of /bin/sed during execution of libtool
> has got to slow it like a sucking chest wound.
> 
> Here is my conjecture...
> Substitution of the code that with potential to cause excessive forking
> along with substitution of snowball code,
> along with substitution of those very expensive case
> statements to detect duplicates could gain 2 magnitudes
> of speed increase on really challenging tasks like koffice.
> At the same time it on platforms that are fork expensive
> could cause an insignificant increase in execution time
> on very light libtool requests that would ordinarlly execute
> quickly anyway.
> 
> If a change in the code makes the best cases as much as
> 10x slower and the worst cases 100x faster then the
> average effect of the change will be faster compilations involving libtool.
> 
> I am not saying that replacement of slow sections of bash code
> along with the potentials for multiple forks to call sed for example
> with a single 3 fork invocation will be faster in every case and every
> platform.
> But it should crush the libtool time spent
> on koffice objects from seconds to miliseconds.
> 
> Do you suppose anyone but me would agree that optimizing libtool
> for best performance on large tasks is better than having libtool
> optimized for best performance on simple tasks and horrifically
> slow performance on large tasks?  Afterall, who cares if it runs
> immeasurably slower on the fastest and easiests tasks if
> knowticible seconds are shaved off the large tasks like koffice?
> I do not care if execution of libtool takes 0.1 seconds or 0.2 seconds.
> However, I do care if execution of libtool takes 0.5 seconds and 25
> seconds.
> Does that make sense?
> 
> libtool is currently optimized for very small and easy tasks.
> But why optimize for the case which is already destined to be fast?
> Instead it aught to be optimized for the large tasks so that compilation
> of a single object file does not require some 2000 invocations of sed,
> right?
> 
> Lemme know if I have a snowball's chance
> in hell of eliminating libtool's snowballing?
> I expect some will adamantly oppose the use
> of grep, sed, sort, uniq in pipes to accomplish
> what can be done in bash with for loops and case statements.
> But if the large libtool tasks are to be speed up by magnitudes
> then the paradigm should be shifted.
> 5 definite forks is better than the possibility
> of no forks with the possibility of hundreds of forks.
> 
> For the simpliest libtool tasks more forks may be executed than before.
> But for the more complex tasks hundreds of forks could be avoided.
> The way it stands now there is a huge disparity between
> libtool execution speed for small and large tasks.
> However, if optimized for large tasks then
> libtool would still execute fast for small tasks,
> and the large tasks would execute only a fraction
> slower than the small tasks since nearly the same
> amount of forks would be executed in both cases.
> 
> The way it is now large tasks not only snowball,
> not only lag across those case statements to locate
> duplicates, but they also incur a huge amount of forks.
> 2000+ invocations of /bin/sed is unreasonable, yes?
> 

- --
Life is short - so eat dessert first!

Eric Blake             address@hidden
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFOhSn84KuGfSFAYARAlJsAJ9OUUSqmxLocb8EDpU6CMLO/A+5aQCePFAL
DYTSXCbUqnLj/Irvj7gROw4=
=x5n3
-----END PGP SIGNATURE-----