testsuite performance (was: [GNU Autoconf 2.59c] testsuite: 75 78 89 102 failed)

[Ralf Wildenhues]

Hi Paul,

* Paul Eggert wrote on Wed, Apr 19, 2006 at 09:32:59PM CEST:
> 
> On a related topic, I tried to reproduced Dalibor Topic's problem on
> Solaris 8 with CONFIG_SHELL=/bin/ksh.  I didn't reproduce it, as
> Solaris 8 uses ksh88 whereas NetBSD evidently uses a clone.

Yes, I think so.  I just reproduced it on the same system as Dalibor
did -- the HP testdrive systems are open for OSS after registration.

> However,
> I discovered that "make check" is incredibly slow -- and this is even
> after I fixed the /bin/sh -n bug.  I think this is because ksh88 still
> creates lots of files in /tmp for here documents.  I couldn't wait for
> it to finish, but I expect "make check" would take about 10 hours.

Hey, some of my test runs took days (on AIX, without setting
CONFIG_SHELL; I believe I reported that at one point).

> Even when I reverted to the default, which uses /bin/bash, "make
> check" still takes about an hour on my 1 GHz 4-CPU sparc server.  (I'm
> estimating this -- I don't have the patience to wait for it to
> finish.)  This is so long, that I expect that people won't bother to
> run it on traditional Unix hosts.  We therefore won't get test results
> back.  This is not a good thing.

Well, if they care about those hosts, then I hope they still try to run
the test suite.

But frankly, I was hoping people would test Autoconf-2.59d on their own
codes.  That's the part where ultimately, we cannot test well.  I was
also hoping the test round I already did could account for a significant
part of our system base (in some measure, at least).

> Part of the problem is that each call to configure does the search for
> the right shell to use.  This is an expensive search.  It should be
> cheaper.  And we shouldn't do it for every test case: one test is
> enough.  I think this severe performance bug is worth fixing before
> 2.60 comes out.

It's not too hard to fix: prepend each ./configure invocation with
$CONFIG_SHELL.  The `./testsuite' startup call should have exported
that, if necessary.

But as noted before: it will reduce some exposure, too.

> Part of the problem is that Sun's /usr/bin/perl does globbing of
> PATTERN by forking and running "sh -c /usr/bin/csh -cf 'set nonomatch;
> glob PATTERN'" (yup, that's how Perl historically did things, and perl
> 5.005_03 fits the mold.  I wish I'd advised Larry Wall way back when
> that this was a dumb idea!).  And csh starts up by doing the
> equivalent of pwd, which walks up the file tree.  This performance
> black hole is fixed in Solaris 10 (which uses perl 5.8.4), but not in
> Solaris 9 (which uses 5.6.1).  I'm not sure we can do much about this,
> except perhaps by avoiding globbing in our Perl scripts whenever
> possible -- i.e., use the shell to glob, not Perl.

Have you measured that this is a significant problem?  (Since my first
bottleneck search in Libtool I've given up trusting only intuition with
shell scripts and the like for finding performance issues.  Who would
have thought the construct
  for item in $lots_of_items; do
    list="$list $item"
  do

could be a _major_ part of the performance hit for `./libtool
--mode=link' with many object files?  Even with GNU bash?)

> Part of the problem is that GNU m4 also does its fair share of
> creating and removing lots of silly little files in /tmp.
> 
> But the main problem is that Autoconf itself is too slow, and it has
> too many tests.

I still maintain that many subtle performance hits may stem from macros
written in a suboptimal way.  But alas, I can't prove it yet.

> One possible workaround is to throw out 90% of the tests for most
> people, reserving the other tests for the maintainers.  This will make
> "make check" ten times faster, and then it will be more likely to get
> used.  This might be worth doing before 2.60 comes out.  As things
> stand, "make check" simply won't get used, at least not by most
> people.

It would help to implement the $CONFIG_SHELL thing (or others mentioned
above) and measure the impact.

OTOH, the Libtool test suite takes very long, too; I've never had the
impression that people don't use it _because of that_, though.  Surely
it's not desirable.

Cheers,
Ralf