Re: Is Autotest the right solution?

[Ralf Wildenhues]

Hi Nathaniel,

* Nathaniel D. Sizemore wrote on Fri, May 19, 2006 at 01:55:53AM CEST:
> 
> Specifically, the test suite needs to do the following:
> 
> 1) Run nightly as a cron job to test the entire package
> 2) Allow developers to easily create and and new tests
> 3) Serve as an example of running the software for end-users
> 
> Unfortunately, some of the requested features have proven difficult  
> to implement using Autotest, particularly that of being an example to  
> end-users.  The issues I've encountered include:
> 
> 1) Naming of test subdirectories.

> 2) Hierarchy of tests.

I can well understand the desire to have these more flexible.  Alas, the
current Autotest cannot achieve that.  What we use instead are keywords,
and combining them to get some flexibility in choosing tests.  (Note
there are some bugs fixed in 2.59c over 2.59 wrt. multiple keywords
selection, and some new features, which allow to combine keywords with
more flexibility.)

> 3) Serving as example to users.  In addition to having cron test the  
> software nightly, we want the tests to show novice users how to use  
> the software.  Due to the complicated nature of the code, this  
> involves several steps, including creating an input file, running the  
> software itself (using some scripts to deal with serial vs. parallel  
> machines, etc.), and processing the results in some fashion (usually  
> by calling external applications).  Currently, I do these steps in a  
> series of AT_CHECK macros, but these are difficult to read for the  
> average user or developer.

Agreed.  The M4 input seems daunting for users as well as casual
contributors.

> Additionally, users need a script that  
> they can run completely independently of the regression tests, unlike  
> the 'run' script created by Autotest that is tied into the test  
> suite.  In other words, if a user is working on a problem of type  
> foo, we'd like to point them to the regression test foo, which would  
> show them how they should go about setting up and running their problem.

This is a problem.  But frankly, I don't see how any sort of
sophisticated test suite solves that: the simplest does that best,
e.g., using Automake's TESTS variable with some shell scripts.

> 4) Developing in testsuite.dir.  Developers have requested the  
> ability to debug and develop new tests in testsuite.dir itself.  That  
> is, editing an input file, and then re-running that single test  
> again.  Currently, the solution is to have an AT_CHECK command that  
> creates a symlink to the input file so it can be edited.  If the test  
> fails, the symlink allows a developer to edit the input, and they can  
> then re-run the test with the 'run' script created by Autotest.   
> However, this removes and re-creates the directory, meaning they have  
> to do a 'cd .' to refresh the filesystem handle.  My dev team hated  
> this.

Hmm, maybe something may be improved here.  I have two ideas in mind:
First, we could try to not remove the directory, only its contents.
This wouldn't be very difficult to realize, but it won't help you much
with developing in there.

Second, I have a vague idea about some sort of notion in a test where it
may be restarted.  Say, for example, your test has a setup phase, where
some data is initialized, and an actual test phase, where this data is
used.  If you specify the point in between, and give the `run' script
some flag to only start off there, it could inhibit cleaning of the
directory and inhibit the setup phase.

Surely, this won't work with destructive tests that modify the input to
previous AT_CHECK commands in further such commands.  But that would be
the responsibility of the developer: to only specify such points if the
restart there would be safe.

I'm not sure if multiple such points are realizable, but ATM I don't see
why not.

Would this be a big benefit for you?

> With the exception of the directory/hierarchy naming, I think all of  
> these problems can be worked around, mostly by creating shell  
> scripts.  These would be user-readable and called by Autotest.   
> However, doing that I lose benefits of using Autotest in the first  
> place, that of testing the return code of each step, being able to  
> run extra commands on success or failure, and auto-generated,  
> detailed log files.

Yes, I agree that this would defeat some of the purpose of using
Autotest.

Thanks for your feedback!  I'm sure others will have more, valuable
comments on it.  Maybe they help making Autotest even more useful.

Cheers,
Ralf