Re: summarize misbehaving tests

[William Pursell]

Ralf Wildenhues wrote:
> Hi William,
> 
> * William Pursell wrote on Fri, Jan 30, 2009 at 08:02:58AM CET:
>> I've written a simple patch that lists the tests that
>> misbehaved (failed or unexpectedly passed), one
>> per line, in the banner.
> 
>> Do you think it would be
>> appropriate to make that functionality a run time
>> environment option, an automake option (like color-tests),
>> or just leave it in for all invocations of make check?
>> (or do you think it's a bad idea?)
> 
> Well, I dearly hope to get to the parallel-tests patches before too
> long, and finishe their issues and push them.  They will (at least when
> the corresponding automake option is used then) make for somewhat
> different (and better IMHO) output (see discussion on automake-patches a
> while ago).  But that doesn't mean you shouldn't show your patch, having
> more choices can only help.
> 

Well, I haven't made any tests, but here's what I'm using so far:
(Not moving this to automake-patches because this isn't really
suitable for inclusion, (no tests))

diff --git a/lib/am/check.am b/lib/am/check.am
index ca400fe..167aa34 100644
--- a/lib/am/check.am
+++ b/lib/am/check.am
@@ -58,6 +58,7 @@ check-TESTS: $(TESTS)
              case " $(XFAIL_TESTS) " in \
              *[\ \     ]$$tst[\ \      ]*) \
                xpass=`expr $$xpass + 1`; \
+               pass_list="$$pass_list $$tst" \
                failed=`expr $$failed + 1`; \
                col=$$red; res=XPASS; \
              ;; \
@@ -74,6 +75,7 @@ check-TESTS: $(TESTS)
                col=$$lgn; res=XFAIL; \
              ;; \
              *) \
+               fail_list="$$fail_list $$tst" \
                failed=`expr $$failed + 1`; \
                col=$$red; res=FAIL; \
              ;; \
@@ -102,10 +104,10 @@ check-TESTS: $(TESTS)
            fi; \
          else \
            if test "$$xpass" -eq 0; then \
-             banner="$$failed of $$all $$tests failed"; \
+             banner="$$failed of $$all $$tests failed:"; \
            else \
              if test "$$xpass" -eq 1; then passes=pass; else passes=passes; 
fi; \
-             banner="$$failed of $$all $$tests did not behave as expected 
($$xpass unexpected $$passes)"; \
+             banner="$$failed of $$all $$tests did not behave as expected 
($$xpass unexpected $$passes):"; \
            fi; \
          fi; \
 ## DASHES should contain the largest line of the banner.
@@ -126,6 +128,10 @@ check-TESTS: $(TESTS)
            test `echo "$$report" | wc -c` -le `echo "$$banner" | wc -c` || \
              dashes="$$report"; \
          fi; \
+         for tst in "$$pass_list $$fail_list"; do \
+           test `echo "    FAIL: $$tst" | wc -c` -le `echo "$$dashes" | wc -c` 
|| \
+             dashes="    FAIL: $$tst"; \
+         done; \
          dashes=`echo "$$dashes" | sed s/./=/g`; \
          if test "$$failed" -eq 0; then \
            echo "$$grn$$dashes"; \
@@ -133,6 +139,12 @@ check-TESTS: $(TESTS)
            echo "$$red$$dashes"; \
          fi; \
          echo "$$banner"; \
+         for tst in "$$fail_list"; do \
+           test -z "$$tst" || echo "    FAIL: $$tst"; \
+         done; \
+         for tst in "$$pass_list"; do \
+           test -z "$$tst" || echo "    PASS: $$tst"; \
+         done; \
          test -z "$$skipped" || echo "$$skipped"; \
          test -z "$$report" || echo "$$report"; \
          echo "$$dashes$$std"; \



-- 
William Pursell