[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: How to abort a test?
From: |
Pedro Alves |
Subject: |
Re: How to abort a test? |
Date: |
Thu, 14 Jan 2016 17:28:26 +0000 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 |
On 01/14/2016 05:13 PM, Simon Marchi wrote:
> On 16-01-14 11:25 AM, Pedro Alves wrote:
>>> The test will be aborted, runtest will output a detailed error, but the
>>> test will still
>>> pass. Intuitively, I would think that a test that throws an error should
>>> automatically
>>> be failed or unresolved, since something unexpected happened.
>>
>> IIUC, you wouldn't want pass/fail to even be reached, so I don't understand
>> what test would you want to fail.
>
> I think unresolved would be appropriate in that case, since it's not the
> program under test
> failed, but the test, or its setup/environment.
OK. I'd say "testcase" to disambiguate with a particular pass/fail invocation.
>
> According to the Dejagnu doc, unresolved: "... usually means the test did not
> execute as
> expected, and a human being must go over results to determine if it passed or
> failed (and
> to improve the test case). ". I think that applies here.
>
> So when there is an uncaught exception coming from the test, Dejagnu could
> issue a:
>
> unresolved "Uncaught error from test " # needs a better message
>
> The important part is that the runtest command returns a failure return code,
> so that
> automated builds or scripts consider the run as failed. To me, a test that
> ends with
> an exception is not a test that ran successfully.
>
>>>
>>> The only option I see right now would be to fix the whole return chain and
>>> add proper
>>> error handling everywhere, to exit early when an error happens. However,
>>> that means
>>> changing tens (hundreds?) of callsites through the testsuite, which is why
>>> I'm
>>> looking for alternative solutions first.
>>
>> Test usually return early if running to main fails (if ![runto_main]), so
>> maybe it
>> won't be that many places. Maybe some judiciously placed "return -code
>> return"
>> hacks saves you a good chunk. I don't have any other idea.
>
> Many tests use gdb_run_cmd directly without checking the result:
>
> gdb/testsuite $ grep -rin '^gdb_run_cmd$' gdb.* | wc -l
> 73
>
> So they would need to be changed:
>
> -gdb_run_cmd
> +if ![gdb_run_cmd] {
> + fail "Failed to run"
> +}
>
That's where the "return -code return" hack would come in. You'd
do that inside gdb_run_cmd so that would return directly to gdb_run_cmd's
caller. But it's an ugly hack.
>> For the particular case of gdbserver not being present in the target,
>> probably the easiest would be to check that earlier, likely before
>> runtest, even. Not ideal, since the testsuite can mix native and gdbserver
>> tests, for instance, but...
>
> And it's a bit hard to check in the case of a remote target, given that it's
> runtest
> that knows how to "compute" the remote hostname, username, expected gdbserver
> path,
> etc, from the --target_board.
I meant, before runtest, the dejagnu procedure, not before invoking the
runtest binary. E.g., straight from the target board file, say.
E.g., with:
--- c/gdb/testsuite/boards/gdbserver-base.exp
+++ w/gdb/testsuite/boards/gdbserver-base.exp
@@ -33,6 +33,8 @@ set_board_info gdb,predefined_tsv "\\\$trace_timestamp"
set GDBFLAGS "${GDBFLAGS} -ex \"set auto-connect-native-target off\""
+error "gdbserver not present on target!"
+
proc ${board}_file { dest op args } {
if { $op == "delete" } {
return 0
then I get:
$ time make check RUNTESTFLAGS="--target_board=native-gdbserver"
make[1]: Entering directory `/home/pedro/gdb/mygit/build/gdb/testsuite'
Nothing to be done for all...
make check-single
make[2]: Entering directory `/home/pedro/gdb/mygit/build/gdb/testsuite'
rootme=`pwd`; export rootme; srcdir=/home/pedro/gdb/mygit/src/gdb/testsuite ;
export srcdir ; EXPECT=`if [ "${READ1}" != "" ] ; then echo
${rootme}/expect-read1; elif [ -f ${rootme}/../../expect/expect ] ; then echo
${rootme}/../../expect/expect ; else echo expect ; fi` ; export EXPECT ;
EXEEXT= ; export EXEEXT ;
LD_LIBRARY_PATH=$rootme/../../expect:$rootme/../../libstdc++:$rootme/../../tk/unix:$rootme/../../tcl/unix:$rootme/../../bfd:$rootme/../../opcodes:$LD_LIBRARY_PATH;
export LD_LIBRARY_PATH; if [ -f ${rootme}/../../expect/expect ] ; then
TCL_LIBRARY=${srcdir}/../../tcl/library ; export TCL_LIBRARY ; fi ; runtest
--target_board=native-gdbserver
Test Run By pedro on Thu Jan 14 17:24:37 2016
Native configuration is x86_64-pc-linux-gnu
=== gdb tests ===
Schedule of variations:
native-gdbserver
Running target native-gdbserver
Using
/home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/native-gdbserver.exp
as board description file for target.
Using
/home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp as
board description file for target.
ERROR: tcl error sourcing board description file for target
/home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp.
gdbserver not present on target!
gdbserver not present on target!
while executing
"error "gdbserver not present on target!""
(file
"/home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp"
line 36)
invoked from within
"source
/home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp"
("uplevel" body line 1)
invoked from within
"uplevel #0 source
/home/pedro/gdb/mygit/src/gdb/testsuite/boards/../boards/gdbserver-base.exp"
invoked from within
"catch "uplevel #0 source ${dir}/${initfile}" error"
make[2]: *** [check-single] Error 1
make[2]: Leaving directory `/home/pedro/gdb/mygit/build/gdb/testsuite'
make[1]: *** [check] Error 2
make[1]: Leaving directory `/home/pedro/gdb/mygit/build/gdb/testsuite'
make: *** [check] Error 2
real 0m0.404s
user 0m0.336s
sys 0m0.060s
Obviously not ideal; I'm just pointing out the likely easiest.
Thanks,
Pedro Alves
- How to abort a test?, Simon Marchi, 2016/01/12
- Re: How to abort a test?, Pedro Alves, 2016/01/14
- Re: How to abort a test?, Simon Marchi, 2016/01/14
- Re: How to abort a test?,
Pedro Alves <=
- Re: How to abort a test?, Simon Marchi, 2016/01/14
- Re: How to abort a test?, Pedro Alves, 2016/01/14
- Re: How to abort a test?, Simon Marchi, 2016/01/15
- Re: How to abort a test?, Ben Elliston, 2016/01/15
- Re: How to abort a test?, Joel Brobecker, 2016/01/17
- Re: How to abort a test?, Simon Marchi, 2016/01/18
- Re: How to abort a test?, Joel Brobecker, 2016/01/21