Re: Only ~50 error messages get processed,

[Jack Howarth]

Note that this issue seems to only be fixable from within dejagnu and
that the commit of 6d2e2d3791bcea70131a6cf64a0a5223333a7b8e
effectively broke the log_summary proc in dejagnu's lib/framework.exp
by reseting the testcnt global which is used as follows...

    # If the tool set `testcnt', it wants us to do a sanity check on the
    # total count, so compare the reported number of testcases with the
    # expected number.  Maintaining an accurate count in `testcnt' isn't easy
    # so it's not clear how often this will be used.
    if {[info exists testcnt]} {
        if { $testcnt > 0 } {
            set totlcnt 0
            # total all the testcases reported
            foreach x { FAIL PASS XFAIL KFAIL XPASS KPASS UNTESTED
UNRESOLVED UNSUPPORTED } {
                incr totlcnt test_counts($x,$which)
            }
            set testcnt test_counts(total,$which)

            if { $testcnt>$totlcnt || $testcnt<$totlcnt } {
                if { $testcnt > $totlcnt } {
                    set mismatch "unreported  [expr {$testcnt - $totlcnt}]"
                }
                if { $testcnt < $totlcnt } {
                    set mismatch "misreported [expr {$totlcnt - $testcnt}]"
                }
            } else {
                verbose "# of testcases run         $testcnt"
            }

            if {[info exists mismatch]} {
                clone_output "### ERROR: totals do not equal number of
testcases run"
                clone_output "### ERROR: # of testcases expected    $testcnt"
                clone_output "### ERROR: # of testcases reported    $totlcnt"
                clone_output "### ERROR: # of testcases $mismatch\n"
            }
        }
    }
    foreach x { PASS FAIL XPASS XFAIL KPASS KFAIL UNRESOLVED UNTESTED
UNSUPPORTED } {
        set val $test_counts($x,$which)
        if { $val > 0 } {
            set mess "# of $test_counts($x,name)"
            if { $xml } {
                xml_output "  <summary>"
                xml_output "    <result>$x</result>"
                xml_output "    <description>$mess</description>"
                xml_output "    <total>$val</total>"
                xml_output "  </summary>"
            }
            if { [string length $mess] < 24 } {
                append mess "\t"
            }
            clone_output "$mess\t$val"
        }

On Sat, Mar 28, 2015 at 4:58 PM, Jack Howarth
<address@hidden> wrote:
> The breakage from commit 6d2e2d3791bcea70131a6cf64a0a5223333a7b8e is
> due to execution of...
>
>     foreach x $test_names {
>         set test_counts($x,count) 0
>     }
>
> in proc runtest introduced by the usage of reset_vars. I can confirm that...
>
> diff -uNr dejagnu-1.5.2.orig/runtest.exp dejagnu-1.5.2/runtest.exp
> --- dejagnu-1.5.2.orig/runtest.exp 2015-02-03 21:25:26.000000000 -0500
> +++ dejagnu-1.5.2/runtest.exp 2015-03-28 16:48:10.000000000 -0400
> @@ -1441,13 +1441,22 @@
>  #
>
>  proc runtest { test_file_name } {
> +    global test_names test_counts
> +    global prms_id
> +    global bug_id
>      global test_result
>      global errcnt
>      global errorInfo
>      global tool
>
>      clone_output "Running $test_file_name ..."
> -    reset_vars
> +    set prms_id 0
> +    set bug_id 0
> +
> +    foreach x $test_names {
> +        set test_counts($x,count) 0
> +    }
> +
>      set test_result ""
>
>      if {[file exists $test_file_name]} {
>
> reproduces the the breakage.
>
>
> On Sat, Mar 28, 2015 at 3:20 PM, Jack Howarth
> <address@hidden> wrote:
>> Okay, a regression hunt of dejagnu master git revealed that this
>> regression is actually quite old and that the offending commit was...
>>
>> http://git.savannah.gnu.org/cgit/dejagnu.git/commit/?id=6d2e2d3791bcea70131a6cf64a0a5223333a7b8e
>>
>> 2011-03-14 Jan Kratochvil <address@hidden>
>>
>>  * runtest.exp (runtest): Call reset_vars at initialisation so that
>>   a perror does not affect test results in other test scripts.
>>
>>
>> This issue was latent in 1.5.1 because the 1.5 branch was reused for
>> developing 1.5.1 whereas 1.5.2 was created from master instead.
>> I can confirm that reverting this change in 1.5.2 eliminates the
>> problem and I get the expected...
>>
>> === obj-c++ Summary for unix/-m32 ===
>>
>> # of expected passes 3069
>> # of unexpected failures 39
>> # of expected failures 2
>> # of unresolved testcases 57
>> # of unsupported tests 68
>>
>> === obj-c++ Summary for unix/-m64 ===
>>
>> # of expected passes 3002
>> # of unexpected failures 45
>> # of expected failures 18
>> # of unresolved testcases 60
>> # of unsupported tests 74
>>
>> === obj-c++ Summary ===
>>
>> # of expected passes 6071
>> # of unexpected failures 84
>> # of expected failures 20
>> # of unresolved testcases 117
>> # of unsupported tests 142
>>
>> On Sat, Mar 28, 2015 at 1:51 PM, Jack Howarth
>> <address@hidden> wrote:
>>> Just to clarify. I am looking at
>>> http://git.savannah.gnu.org/cgit/dejagnu.git which shows branches for
>>> dejagnu-1.5 which behaves like 1.5.1 is I do a hard reset to the
>>> 2013-03-08 commit which marked the ChangeLog to 1.5.1. The trouble is
>>> that there is no matching dejagnu-1.5.1 branch but only a
>>> dejagnu-1.5.2 branch. The 1.5.2 branch is problematic as I can't seem
>>> to find a commit in that branch that behaves like 1.5.1. One would
>>> think at going back to..
>>>
>>> http://git.savannah.gnu.org/cgit/dejagnu.git/commit/?h=dejagnu-1.5.2&id=33b71e46c0a27fee374a922eb0792b9c618c54a8
>>>
>>> would be just like 1.5.1 but it doesn't. Any idea when the 1.5.2
>>> branch was created?
>>>
>>> On Sat, Mar 28, 2015 at 1:32 PM, Jack Howarth
>>> <address@hidden> wrote:
>>>> I am having problems trying to regression hunt this issue in dejagnu
>>>> git. Using the gcc test suite run...
>>>>
>>>> make -k check-obj-c++ RUNTESTFLAGS="--target_board=unix'{-m32,-m64}'"
>>>>
>>>> I get...
>>>>
>>>> dejagnu 1.5.1
>>>>
>>>>
>>>>                 === obj-c++ Summary for unix/-m32 ===
>>>>
>>>> # of expected passes            3069
>>>> # of unexpected failures        39
>>>> # of expected failures          2
>>>> # of unresolved testcases       57
>>>> # of unsupported tests          68
>>>>
>>>>                 === obj-c++ Summary for unix/-m64 ===
>>>>
>>>> # of expected passes            3002
>>>> # of unexpected failures        45
>>>> # of expected failures          18
>>>> # of unresolved testcases       60
>>>> # of unsupported tests          74
>>>>
>>>>                 === obj-c++ Summary ===
>>>>
>>>> # of expected passes            6071
>>>> # of unexpected failures        84
>>>> # of expected failures          20
>>>> # of unresolved testcases       117
>>>> # of unsupported tests          142
>>>>
>>>> but...
>>>>
>>>> dejagnu 1.5.2
>>>>
>>>>                 === obj-c++ Summary for unix/-m32 ===
>>>>
>>>> # of expected passes            128
>>>>
>>>>                 === obj-c++ Summary for unix/-m64 ===
>>>>
>>>> # of expected passes            128
>>>>
>>>>                 === obj-c++ Summary ===
>>>>
>>>> # of expected passes            6071
>>>> # of unexpected failures        84
>>>> # of expected failures          20
>>>> # of unresolved testcases       117
>>>> # of unsupported tests          142
>>>>
>>>> I can reproduce the dejagnu 1.5.1 results with a build using...
>>>>
>>>> git clone git://git.sv.gnu.org/dejagnu.git
>>>> cd dejagnu
>>>> git reset --hard 6bd8dddc81d8a5f2e835f148a9be1509f3edb147
>>>>
>>>>
>>>> which is the parent commit for
>>>> http://git.savannah.gnu.org/cgit/dejagnu.git/commit/?id=52dad3fd75c64d8e8cc0de5f8ca123c4aa7c32d2
>>>>
>>>> however if try the same for the first commit in the gcc-1.5.2 branch...
>>>>
>>>> http://git.savannah.gnu.org/cgit/dejagnu.git/commit/?h=dejagnu-1.5.2&id=d8912f7d0567831f425115eaeb6ee915e8ea5b21
>>>>
>>>> using...
>>>>
>>>> git reset --hard 33b71e46c0a27fee374a922eb0792b9c618c54a8
>>>>
>>>> the build acts like 1.5.2 even though it should only have one
>>>> additional commit compared to the 1.5.1 release. Any clarifications on
>>>> how these git trees are arranged would be appreciated. So far I am
>>>> having no luck walking back in trunk to a point where the 1.5.1
>>>> behavior can be reproduced.
>>>>           Jack