Re: quiet mode for make check (silence passing tests, list failing tests)

[Chris Pickett]

Hi Ralf and Akim and co.,

(Please include me in any reply, I am not subscribed.)

Ralf Wildenhues wrote:
> * Chris Pickett wrote on Wed, Jan 23, 2008 at 10:56:29AM CET:
> > I have one test program for each module in my library.  Each writes to 
> > stdout and stderr.  Since I have many tests, this generates a lot of 
> > output when I run `make check'.  I would like to change `make check' so 
> > that it:
> >
> > 1) Causes only those tests that fail to print anything
> > 2) Additionally lists the tests that fail at the end
> But it's also possible inside Makefile.am.  In fact, there have been
> patches floating around for parallel testsuites, which can hopefully be
> integrated before 1.11, which deal smartly with verbose output (keeping
> verbose log files of failed tests).

I found the check.mk patch (set of redefinitions) you were referring to:

http://www.mail-archive.com/address@hidden/msg12373.html

The original is available here:

https://trac.lrde.org/vaucanson/browser/trunk/build-aux/check.mk

I adapted it for my own project here:

https://svn.sable.mcgill.ca/sable/spmt/libspmt/build-aux/check.mk

These are the differences in order if you diff the two files (comments 
are painful with trailing backslashes):

1) Remove a blank line for each individual test log:

FAIL: <test>
============
              <--- here
Output from test.

2) Don't print PASS: <test> or FAIL: <test> for each test as it 
executes.  This is just too verbose for me; if everything passes, all I 
want is a green bar (box) at the bottom.  If something fails, I want the 
full log, not just a message that it failed.

3) Don't show the exit status for a test.  Too verbose for me.

4) Remove the boilerplate echoing the PACKAGE_STRING and the name of the 
file into $(TEST_SUITE_LOG) since this is obvious from position.  Also 
remove some idiosyncratic formatting stuff (".. contents:: :depth: 2").

5) Remove the "See $(subdir)/$(TEST_SUITE_LOG)" message (see change #7). 
 I want to keep the red bar (box) message concise.

6) s/Please report it to/Please report to/ in line with current Automake.

7) Always send $(TEST_SUITE_LOG) to stdout.  If a test fails, I'm going 
to check what happened anyways.  Originally, a VERBOSE variable guarded 
that, but I think VERBOSE should be default.

8) Show $(TEST_SUITE_LOG) before the am__text_box announcing the total 
number of tests that pass/fail.  I think you want the red/green bar to 
appear at the bottom of the output and not have to scroll back for it.

9) Always remove $(TEST_SUITE_LOG) so that the pass/fail output is 
always regenerated.  This does not significantly affect running time.

Additionally, in order to use this check.mk, I added the following to my 
tests/Makefile.am:



# this rule seems necessary to quiet errors
# about check.mk not knowing how to create .log
.log:

LAZY_TEST_SUITE = 1
TEST_LOGS = $(TESTS:=.log)

# parallel/quiet make check from vaucanson-1.1.1
include $(top_srcdir)/build-aux/check.mk




I don't really know about the .log rule but meh, it works.  Also, I 
tried using the EXTRA_PROGRAMS variable, and while it apparently let the 
tests run concurrently with the compilations, I prefer the output I get 
with make check (`<test> is up to date.') and also since change #2 
removes the immediate feedback on pass/fail I still have to wait for 
everything to run anyway.  Trying to look for a red message and hit 
Ctrl-c and then cat the appropriate tests/<test>.log is not what I want.

Anyway, thanks Akim for the hard work on check.mk.  It's much much 
better for me.  By the way, my output looks like this:

gmake  check-TESTS
gmake[2]: Entering directory `$HOME/libspmt/tests'
rm -f test-suite.log
gmake[3]: Entering directory `$HOME/libspmt/tests'

FAIL: predictor
===============
Running suite(s): predictor
Assertion failed: predictor != NULL, file  predictor.c, line 127
66%: Checks: 3, Failures: 1, Errors: 0
predictor.c:46:F:core:check_half_width_return_value:0: Assertion 
'(spmt_predictor_predict (predictor) == 3)' failed

==================================================
 1 of 48 tests failed
 Please report to chris.pickett AT mail.mcgill.ca
==================================================

gmake[3]: *** [test-suite.log] Error 1
gmake[3]: Leaving directory `$HOME/libspmt/tests'
gmake[2]: *** [check-TESTS] Error 2
gmake[2]: Leaving directory `$HOME/libspmt/tests'
gmake[1]: *** [check-am] Error 2
gmake[1]: Leaving directory `$HOME/libspmt/tests'
gmake: *** [check-recursive] Error 1

where the box is in red.  In case you're wondering, I'm using the Check 
unit testing framework for C.  It works quite well for me, but I'm also 
a maintainer.  When things do work, the output looks like this now:

make  check-TESTS
rm -f test-suite.log

=====================
 All 48 tests passed
=====================

where the box is in green.  That's on a different machine (OS X, not 
AIX), and for some reason I don't get the entering/leaving messages. 
Not sure which I prefer.  Of course, all the .log files are available 
for individual tests, regardless of pass/fail status.

Finally, I noticed that check.mk is GPL, where as the Autoconf stuff in 
build-aux comes with this notice:

# As a special exception to the GNU General Public License, if you 



# distribute this file as part of a program that contains a 



# configuration script generated by Autoconf, you may include it under 



# the same distribution terms that you use for the rest of that program.

I don't know if it matters.  I'm assuming that using check.mk as GPL 
doesn't mean that my library is forced to be GPL:

## This file comes from vaucanson-1.1.1.  Since it only affects the
## build process, it is assumed that libspmt is not forced to be GPL.

but I guess I would feel better if that exception notice was there? 
(That `?' has a mysterious meaning.)  I suppose if check.mk gets into 
Automake (which I hope) then everything will straighten out anyway.

Cheers,
Chris