Re: unit testing with autotools

[Thomas Porschberg]

Am Tue, 14 Mar 2006 11:32:31 -0500
schrieb Chris Pickett <address@hidden>:

> [[ Sorry, I originally sent this from an unsubscribed account ]]
> 
> -------- Original Message --------
> Subject: Re: unit testing with autotools
> Date: Tue, 14 Mar 2006 11:25:06 -0500
> From: Chris Pickett <address@hidden>
> Reply-To: address@hidden
> To: Ralf Wildenhues <address@hidden>
> CC: address@hidden
> References: <address@hidden> 
> <address@hidden>
> 
> Hi Ralf,
> 
> Ralf Wildenhues wrote:
> > Hi Chris,
> > 
> > * Chris Pickett wrote on Mon, Mar 13, 2006 at 06:13:07PM CET:
> > 
> >>I'd like to do unit testing for test-driven development of C
> >>programs. I'd also like to use up-to-date autotools.  I was
> >>wondering if anyone had any experience with unit testing in their
> >>projects and had recommendations, or even if autoconf was
> >>considering it for the future.
> >>
> >>By unit testing I mean writing multiple small tests for each
> >>function in an interface.  The prescribed development methodology
> >>is to write failing tests before you write code that will make them
> >>pass.
> >>
> >>Autotest would be good as a wrapper around a unit-testing
> >>framework, but not good for driving the actual low-level tests.
> >>The same goes for DejaGNU.
> > 
> > 
> > Out of curiosity: why exactly do Autotest and DejaGNU not qualify
> > for this job?  Which parts are missing?
> 
> I'm sorry, I should have explained it better.  But first a
> disclaimer: I only started thinking about unit testing on Friday, and
> I only came to understand the basic autotools setup for starting a
> project from scratch over the weekend, although in the past I have
> used autotools and m4 a reasonable amount, but starting from other's
> work.  So I may completely wrong here.
> 
> Autotest and DejaGNU are great for "system testing" -- running
> compiled programs and checking exit success, failure, the correctness
> of stdout and stderr, handling of command line options, stuff like
> that.
> 
> However, there isn't any support for checking the correctness of
> individual functions in a module.  Ideally for each function in your
> interface, you have several test functions.  Then if you want to
> refactor your interface implementation at a later date, you already
> have a bunch of test functions to help reassure you that the
> refactoring works.  You have client code for each function from the
> get go.
> 
> The way a unit testing framework handles this is by making it easy to
> create small test functions that exercise your real functions.
> DejaGNU provides an API to such frameworks in dejagnu.h
> 
> The archetypal unit testing framework, xUnit, is described pretty
> clearly here:
> 
> http://www.oreilly.com/catalog/unitest/chapter/ch03.pdf
> 
> It describes the following simple set of classes:
> 
>    -- TestCase, a  test class with several individual test methods
>    -- TestFixture, providing setUp() and tearDown() functions for
> tests -- TestRunner, for running individual tests and reporting
>         speed, success
>    -- TestSuite, which can contain TestCases or even other TestSuites
>    -- TestResult, for accumulating test outcomes
> 
> Some of the additional features that I like in check, the C unit
> testing framework I'm considering:
> 
>    -- Tests can optionally be forked in their own address space
>      -- you can catch signals, expected or unexpected
>      -- you can stop early exits
>      -- you can prevent tests using the same setup data from
>         interfering with each other
>    -- You can have test timeouts
>    -- You can loop through a table of values with a looping test
>    -- Integration with gcov, to report code coverage (percentage of
> code that your tests exercise)
> 
> So, why did I write here?
> 
> 1) to find out if other people had gotten existing unit testing
> frameworks to play nicely with autotools.  Check actually was designed
> around the 2.13 series, and although development has dwindled, it
> probably could work pretty well.  I was just hoping to hear about
> other's experiences before committing to it.
>    -- Braden says he uses Boost Test
>    -- Ed says he doesn't bother with a heavyweight framework and
>      just writes his own API test code by hand
> 
> 2) to find out if people had managed to do it cleanly using just
> Autotest/DejaGNU
> 
> 3) (failing 2) to suggest that making it easy to unit test your code
> might make a good addition to autotools.  I'm not sure about this,
> writing something language-agnostic could be tricky.
> 
> >>I'm also interested in more detailed or "best practice" autotest 
> >>tutorials but the advice in the archives seems to be, "look at
> >>other projects and existing tutorials and cobble something
> >>together."
> > 
> > 
> > FWIW, CVS M4 and CVS Libtool use Autotest.
> 
> I checked out M4 and Libtool and looked around.  I didn't see anything
> testing individual C functions (not really applicable for M4), but
> maybe I missed it.  For example, there's nothing that I saw testing
> the functions in libltdl/ in Libtool, and nothing testing the .c
> files in src/ in M4.  Of course, they do get tested indirectly.
> 
We use cppunit (http://cppunit.sourceforge.net/cppunit-wiki/FrontPage)
together with autotools.

Thomas

--