Severe performance problem and proposed solution

[Zack Weinberg]

It's regularly reported to the GCC list that libstdc++'s configure
script takes several hours to execute on certain proprietary Unixes,
such as alpha-dec-osf4*.  This is due to a bug in the shell.  Some
versions of the Bourne shell implement here documents by writing the
text to a file in /tmp.  These files are not deleted until the shell
exits.  When the shell spawns a subprocess from inside a conditional
block, the child creates a hard link to every single scratch file
which currently exists; when the child exits or execs, it unlinks them
all first.  (No one is precisely sure why this is done, but we are
certain that this is what happens.)

Most of libstdc++-v3's configure script is wrapped in an if ... else
... fi construct; it creates about 1500 here documents and runs at
least twice as many subprocesses.  Therefore, several million calls to
link(2) and unlink(2) occur during the execution.  As every last one
of those causes a synchronous disk write, you can appreciate why it
takes hours to execute.

Richard Henderson and I experimented with using echo instead of here
documents[1].  The trouble with this is of course that echo may
interpret backslashes and leading -, so both have to be escaped.  We
got a two-page gobbet of M4 that sort of worked, but couldn't help at
all with AC_DEFINE_UNQUOTED and the like.  I think this approach is a
dead end.

It is trivial to write a short C program that prints all its arguments
to standard output, separated by spaces, with no interpretation
whatsoever.  Replacing all instances of

cat > conffile <<'EOF'
text here
EOF

with

./acdumbecho 'text here' > conffile

will eliminate the performance problem, and will not have any negative
effect on systems that don't have broken here documents; we've
replaced a spawn of /bin/cat with a spawn of ./acdumbecho, which
should be a performance wash.

I see only two problems with this approach:

- The configure script has to compile acdumbecho to execute on the
_build_ machine.  In a cross-compiling situation, $CC may not be a
native compiler.  I strongly suspect that a functional native compiler
will always be findable (it can even be an old buggy K+R compiler) but
I don't know that for certain.

- We may have problems with command line length limits.  (It is
necessary to replace _all_ uses of here documents, including those for
long test programs, to solve the problem completely.)

Thoughts?

zw

p.s. please cc: me on replies, i'm not subscribed to this list.