|
From: | Bryce McKinlay |
Subject: | Re: Progress on a Classpath mauve suite? |
Date: | Wed, 09 Mar 2005 13:55:48 -0500 |
User-agent: | Mozilla Thunderbird 1.0 (X11/20041206) |
Roman Kennke wrote:
Am Montag, den 07.03.2005, 21:49 -0600 schrieb Archie Cobbs:In this thread: http://lists.gnu.org/archive/html/classpath/2004-12/threads.html#00137 we talked about putting together somehow a list of Mauve tests that all Classpath-based VMs should expect to pass, along with another (complementary) list of those tests a Classpath-based VM should expect to fail.Sorry if I am completely naive here, but why should there be tests, that a Classpath-based VM should be expected to fail?? Isn't the whole pointto (ideally) pass all tests?
Ideally, yes. But in practice, it isn't practical to pass all the tests ;-). What mauve does need is an easier way to highlight regressions. Currently the way people check for regressions is to do before & after mauve runs and diff the output to look for new FAILs, which is obviously pretty tedious. It would be nice if mauve could do this automatically by feeding it a file containing a list of known FAILs for a given VM, so that it would only let you know about failures that were previously expected to pass (and successes that were expected to fail!). This is complicated a little by mauve's design - there isn't a way to uniquely identify each "test" (ie each check() call), and sometimes some check()'s wont be run if a previous one failed, so fixing one bug can sometimes cause more FAILs. This means that it may not be clear if a new FAIL is actually a regression or a check that just wasn't run before. We could make the tests more course-grained from the point of view of the expected-failure machinery - ie track success/failure at the Testlet level, but that has the disadvantage of potentially not showing new regressions within a given a testlet that was already failing.
Bryce
[Prev in Thread] | Current Thread | [Next in Thread] |