Re: test driven development for GNUstep

[Alexander Malmberg]

Adam Fedor wrote:
> > Just loose thoughts but ...
> > We *could* go down the route of writing each testcase as a small ObjC
> > code fragment ... then we
> > wouldn't really need a test framework at all ... just a makefile and
> > the test code.
> 
> I was actually just looking at how gcc does their tests and it appears
> that in Objective-C, they essentially do this and just use DejaGNU to
> compile and run the programs (and perhaps format the output?). We might
> as well write our own simple framwork to do that if that's the route we
> want to go, though.

Which is basically what I've been toying with. I've released a basic
version at:

http://w1.423.telia.com/~u42308495/alex/AlexsGNUstepTests-0.1.tar.gz

To write a new test case, just drop an .m file in any (new or existing)
subdirectory. Each test is compiled and run by itself, so it should
contain a main function that does the testing. Exiting successfully
gives you a 'COMPLETED', exiting with a non-zero exit status (or
crashing, triggering an assertion, or throwing an uncaught exception)
gives a 'FAIL'. If you want to, you can #include "Testing.h" and use the
pass() function to print PASS/FAIL messages for "sub-tests" and always
exit successfully.

./runtests.sh runs all tests (logging to tests.log and tests.sum).
./runtest.sh foo/zot.m runs a specific test.

Not many tests yet, and most of them are auto-generated. I get (comments
added):

     13 COMPILEFAIL (a test failed to compile)
    230 COMPLETED (a test exited successfully)
      3 FAIL (a "sub-test" failed, or a test exited unsuccessfully)
     14 PASS (a "sub-test" passed)

(I have fixes for some of the failures, but I need some failures to test
the test suite. :)

114 lines of shell code, a 16 line header, and a 10 line makefile
template. :) It may not be perfect, but it provides many features that I
think any new test suite framework should provide:

* No external dependencies.

* Tests in plain objective-c. No weird languages, and you don't have to
use any fancy macros or functions to test things. Both lower the barrier
for writing (and reading) tests.

* Easy to add tests. Just drop in the new .m file (and any auxiliary
files).

* Robust. Handles tests that crash or fail to compile.

* It's easy to run a specific test, and (more importantly) it's easy to
debug a specific test. Just run './runtest.sh foo.m' (and then go to the
directory of the test and run 'make'; a --no-clean option will soon
remove the need for this step), and you'll have a test binary in the
directory of that test with just that test in it. Easy to run over and
over again, or run in gdb.

* Doesn't rely on -base, so if -base fails horribly, it will detect the
failures instead of crashing itself.


That's all I can think of now. :) It's a bit slow due to all the
compiling and linking, but that's the price you pay for robustness. I'm
working on an option that will make this better in (what should be :-)
the common case: no catastrophic failures (crashes or compile errors).

> Has anyone thought about how we test GUI programs?

Yes: a custom testing backend. This is necessary to give the tests a
consistent environment (lots of -gui depends on backend details; lots of
things simply aren't possible to test unless you know eg. which fonts
are available). It would also be used to generate synthetic events
sequences (necessary to test event handling parts of -gui), and to dump
the drawing calls (necessary to test that things are drawing/redrawing
themselves correctly).

- Alexander Malmberg