Re: testsuite performance

[Paul Eggert]

Ralf Wildenhues <address@hidden> writes:

> Hey, some of my test runs took days

You're a patient guy!  Not everyone will be that patient, I'm afraid.

> Have you measured that this is a significant problem?

It's hard to measure.  The stuff I was reporting was anecdotal.  Here's
an example output of "ptree":

    25617 make check
      25626 bash -ce failcom='exit 1'; \ for f in x $MAKEFLAGS; do \   case $f 
in \     *=*
        25802 bash -ce failcom='exit 1'; \ for f in x $MAKEFLAGS; do \   case 
$f in \     *=*
          25803 make check
            25821 make check-local
              25830 /bin/bash ./testsuite
                2472  /bin/bash ./testsuite
                  2474  cat
                2471  /bin/bash ./testsuite
                  2475  /bin/bash ./testsuite
                    2476  /usr/bin/perl -w 
/u/cs/fac/eggert/junk/autoconf-2.59d/bin/autoupdate -B /u/cs/f
                      2612  sh -c 
/u/cs/fac/eggert/junk/autoconf-2.59d/tests/autom4te --language=autoconf -
                        2613  /usr/bin/perl -w 
/u/cs/fac/eggert/junk/autoconf-2.59d/bin/autom4te -B /u/cs/fac
                          2680  <defunct>
                          2683  sh -c /usr/bin/csh -cf 'set nonomatch; glob 
/var/run/rpc_door/am4t7535.2613/*' 

That "/var/tmp/run/rpc_door" is because I set
TMPDIR=/var/tmp/run/rpc_door, as this happens to be the only file
system on that host that is mounted on swap.  If I run with the
default TMPDIR, things get slower I'm sure.

I also did a "truss -f -p" for a while, traced 532723 system calls,
analyzed the resulting output with the command
"sed -n '/exec[a-z]*(/{s/.*("//; s/".*//;p;}' | sort | uniq -c | sort -rn",
and got this output:

    356 /usr/bin/expr
    347 /usr/bin/cat
    270 /usr/bin/rm
    149 /usr/bin/sed
    133 /usr/bin/basename
    104 /usr/bin/diff
     80 /bin/sh
     65 /usr/bin/chmod
     63 /usr/bin/uname
     62 /usr/local/SUNWspro/bin/cc
     41 /usr/local/sunStudio-11/SUNWspro/prod/bin/acomp
     41 /usr/bin/mkdir
     40 /u/cs/fac/eggert/junk/bin/m4            [This is GNU m4.]
     39 /usr/xpg4/bin/grep
     39 /usr/bin/grep
     37 /usr/bin/dirname
     31 /bin/bash
     30 /usr/bin/ln
     28 /usr/bin/sparcv9/ls
     28 /bin/uname
     24 /usr/bin/csh
     23 /usr/bin/cp
     21 /usr/bin/sparcv9/sort
     21 /usr/bin/hostname
     15 /usr/bin/rmdir
     14 /usr/ccs/bin/ld
     14 /usr/bin/echo
     13 /usr/bin/mv
      8 /w/fac.01/cs/eggert/junk/autoconf-2.59d/tests/autom4te
      8 /w/fac.01/cs/eggert/junk/autoconf-2.59d/bin/autom4te
      8 ./configure
      7 /usr/convex/getsysinfo
      7 /usr/bin/uniq
      7 /usr/bin/oslevel
      7 /usr/bin/hostinfo
      7 /usr/bin/arch
      7 /bin/universe
      7 /bin/machine
      7 /bin/arch
      7 ./a.out
      4 /w/fac.01/cs/eggert/junk/autoconf-2.59d/tests/autoheader
      4 /w/fac.01/cs/eggert/junk/autoconf-2.59d/tests/autoconf
      4 /w/fac.01/cs/eggert/junk/autoconf-2.59d/bin/autoheader
      4 /w/fac.01/cs/eggert/junk/autoconf-2.59d/bin/autoconf
      4 /usr/bin/find

So, that's 7 programs tested, at the cost of 356 calls to /usr/bin/expr
and 347 calls to /usr/bin/cat.  Surely there's a better way.

> I still maintain that many subtle performance hits may stem from macros
> written in a suboptimal way.

Probably true.  But I think the main problem here is the overhead
of forking, execing, and so forth.

> It would help to implement the $CONFIG_SHELL thing (or others mentioned
> above) and measure the impact.

It didn't help, alas.