Testing approach (was Re: [Gnu-arch-users] more on the merge-fest)

[Misha Dorman]

zander wrote: [some words suggesting the use of XP-style test-first methods, 
including...]
> The point being that its easier to define correct behavior from outside 
> then
> it is by writing the actual code that causes that behavior.

Mark Thomas wrote:
> The "best" way is for two different people to write the code and the test 
> ...
> In the absence of this level of redundancy, writing the code and the test
> suite in either order will be "good enough for government work."

Samuel A. Falvo II wrote:
> I disagree with this, to an extent.  I think the customer of the code is
> responsible for writing what are called 'acceptance tests,' but such an
> author is not going to know how specific data structures work inside the
> program.  The application coder must implement a finer-grain of tests
> than the acceptance tests.

Mark Thomas wrote:
> [test-then-code] encourages "coding to the test suite" where you just write 
> code so that it passes

Samuel A. Falvo II wrote:
> No.  The code will work perfectly, as required by the test suite.  A
> case not considered by the unit tests is, by definition, not part of
> the specification of the software.

The most important thing is that the code and the test be derived 
independently from a specification (whether written down explicitly or not). 
All these methods (code then test, test then code, code & test 
independently) attempt to achieve this.

Obviously having a separate coder and tester (typically, both a developers, 
and they test each others work; we are not talking about dedicated test 
teams here, as Samuel seemed to be) provides the greatest independence, 
though it probably isn't particularly suitable to open source. Open source 
development is predicated on user/devs "scratching an itch" and how many 
peoples itch is "write some test cases for someone elses itch"? (And if it 
isn't scratching an itch, it won't be done as well, because the motivation 
isn't there.)

Code-then-test runs the risk that the test is either omitted (as Mark said) 
or simply written to match the code (set up the inputs, run the code, see 
what it produces, after a brief sanity check say "thats the expected" and 
set up the test expected values to match, rather than deriving the expected 
values from first principles). Obviously an individual developer (Tom, for 
example) may have the self-discipline to "do it properly" but in general the 
temptation to be lazy is strong (especially in commercial contexts, where 
time pressure is stronger, I suspect).

Test-then-code is an attempt to avoid this problem/temptation, as well as a 
recognition that (as zander said) in many cases (though not all) writing a 
test is a good way to codify the required functionality. Of course, the risk 
is then (as Mark noted) that the code is written from the test, which (and 
this is the point that Samuel may have missed) can only be a _partial_ 
codification of the specification (how many test cases do you need to 
_exhaustively_ test even something as simple as A + B?). Test-first 
advocates would say that this risk is lower than the risk of omitted or 
"cheated" tests in code-first approaches.

Note that in theory the end results of code-then-test, test-then-code and 
independent-code-and-test could be identical code and identical tests. In 
practice, the nature of the approach influences how both code and test are 
written and structured, just as individual programmers have their own style 
(c.f. "piss on it to make it smell like Tom")

To echo Tom's comments about "art": there is no "best" approach -- it 
depends on the development team, process, and discipline (both internal and 
external, natural and imposed).

I would suggest that people at least _try_ test-then-code, to see if it 
helps them write good tests.
Then you can make an informed decision, based on the project, the team, the 
quality requirements etc.

--
Misha Dorman

_________________________________________________________________
Express yourself with cool emoticons - download MSN Messenger today! 
http://www.msn.co.uk/messenger