Re: mark expensive tests

[John Wiegley]

>>>>> Michael Albinus <address@hidden> writes:

>> That sounds like a great idea.  I usually define three too: sanity tests,
>> regular tests, extensive tests.

> Do we have an agreement how to handle this? There was no further reaction
> for 3 weeks ...

Yes, we will have three test intensity levels:

  1. Sanity tests just make sure that the environment works, and the whole of
     them should run in <30 seconds on typical hardware. Some shops call these
     'smoke tests', as they are just there to make sure that Emacs can
     function in the most basic ways.

  2. Regular tests do not incur intensive CPU or memory costs, and so can be
     run on any hardware. These should finish on a scale of minutes, likely
     <10 minutes at the most.

  3. Extensive tests are given free reign, and may not even be able to
     complete on systems with CPU or memory constraints. These should feel
     free to take up to an hour, maybe even beyond.

  4. Selective tests are never run automatically, but exist to stress test
     some particular area of the system. These could take days, it doesn't
     really matter what their requirements are.

Each test should be marked, and the test runner should take these into account
and provide options for picking which grade of test (or lower) to be run. This
way I can tell Nix to build and run #2 tests. 'make check' should run #2,
'make fullcheck' should run #3, and there should be individual check targets
for the tests in #4.

I'd especially like the file-notify tests to move to #3, since these are what
consistently bog down my testing environment.

Michael, are these changes you would be interested in making?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

signature.asc
Description: PGP signature