unit-testing Automake (was: [PATCH] Modernize, improve and/or extend tests `colon*.test.)

[Ralf Wildenhues]

* Stefano Lattarini wrote on Thu, Jul 22, 2010 at 01:04:03AM CEST:
> At Wednesday 21 July 2010, Ralf Wildenhues wrote:
> > Well, this is a common dilemma in testing: should the tester be
> > allowed to use insider information or not?  [...]

> > Getting the right balance here is the most non-automatic part of
> > testsuite work.  Ideally, we want the testsuite to be as strict as
> > possible without ever peeking in details which do not matter for
> > the documented API semantics.  So that any change we make to
> > automake.in produces no testsuite failure if the change still
> > provides the right user API.
> Agreed.  This would be more feasible and natural if we had unit tests 
> for automake internals...  but, alas, ATM I don't see any way to 
> introduce such tests without an *extensive* reorganization and 
> refactoring of Automake code, which would probably (surely?) be way 
> too much dangerous and time-consuming.  Oh well.

I'm not sure I follow.  We have a handful of unit tests in the
lib/Automake/tests directory.  What's keeping us from adding more?
Also, iff we have good API coverage in the first place, then a
reorganization shouldn't be dangerous, because the testsuite will
tell us if we have broken anything.  Time-consuming, well, yes.

> > For a related example of this theoretizing: your various patches
> > introduce several greps for error message strings.  I have acked
> > them, mostly because I think it is a good idea, generally.  Now, a
> > while ago I started a patch series to add i18n to Automake.
> Ouch.  I really hoped that Automake could have remained i18n-free
> (yes, I have a strong dislike for i18n, esp. w.r.t. command-line 
> tools: new added complexity to gain so little -- and to lose 
> greppability of error and warning messages in the meantime... sigh).

Actually, the added complexity is fairly low in this case ... at least
for the messages produced by perl code.  The gain for some users is
(hopefully) not little, while I agree that for others, there isn't much
gain.

> > In this particular example, the balance is easy to judge: if the
> > testsuite covers all error messages Automake can generate, we have
> > an easy way to automatically ensure all messages really are
> > annotated for translation if all annotated ones are translated
> > (which gettext can verify for me).
> Sorry, I can't understand what you're saying here...

When you add greps for error messages, I ask myself "does this increase
maintainability or lower it"?  When somebody adds a translation
annotation, I ask myself "do we have a way to ensure all messages are
translated"?  Running the testsuite can verify this, when you save the
output, and if the testsuite does show all error messages.

Cheers,
Ralf