Re: [PATCH 0/4] faster gnulib-tool

[Paolo Bonzini]

Bruno Haible wrote:
> Hello Ralf,
> 
> Thank you for your speedups to gnulib-tool. At first I was, of course,
> excited about the 2x speedup. But when looking at the maintainability
> of the code that you propose, I'm not fine with all of it any more.
> 
> My four objections are:
> 
> 1) You observe that forking programs in a shell script is slow, and
>    therefore propose to use more shell built-ins. The problem with it
>    is that I chose to implement gnulib-tool in shell (for the control
>    structure) and sed (for the text processing).
> 
>    If you want to achieve good speedups for scripts that use 'sed':
>    can you work towards making 'sed' a bash built-in? This is challenging,
>    but if you are after performance, that would be promising.

The only microoptimization that is worthwhile doing to speed up shell
scripts, is to avoid forking.  This is *no exaggeration*.  (I am not
talking about algorithmic improvements; though in some cases, as Ralf
showed, forking can hide the benefits of algorithmic improvements).

We (Ralf, Eric and I) saved over 30% of execution time of Autoconf
scripts on Cygwin, and a few percent on Linux too, just by removing one
or two forks here and there.  It's not about making sed a bash builtin,
it's about not forking for things such as

   (...)
   echo abc | ...

A while ago I made a lot of timings regarding the speed of various shell
constructs; you can find them on the Autoconf list.  Here are the
relevant ones:

   $ time sh -c 'for i in `seq 1 1000`; do :; done'
   user    0m0.034s
   sys     0m0.024s

   $ time sh -c 'for i in `seq 1 1000`; do (:); done'
   user    0m0.486s
   sys     0m2.377s

   $ time sh -c 'for i in `seq 1 1000`; do echo abc | :; done'
   user    0m0.958s
   sys     0m4.657s

echo and : are shell builtins, but they fork, so they're slow.  s/:/sed/
and you see my point.

If this 10x-30x improvement affected 20% of the shell execution time,
one could expect a decent speedup.

That would probably amount to a rewrite of bash, dash, or whatever else.
   You would have to make the main shell loop centered around
event-driven processing of file descriptors, to provide all the pipes
with a single process.

A fun project, but probably not one that I or anyone else will attempt
without funding. :-)

Paolo