Re: Run (some) tests more automagically?

[Lars Ingebrigtsen]

Pip Cet <pipcet@gmail.com> writes:

> Yes, let's! Except we need to agree on what "has been changed" means.
> My initial idea would be to create a time stamp file the first time
> make is run in an emacs directory and consider only those source files
> which are newer than the time stamp, only if they're recompiled.

Hm, yes, a time stamp would probably be the way to go...

> But what should we do when we run "git pull"? Should the time stamp be
> updated for all tests after every make, for successful tests only, or
> only if all tests that were run were successful? Should the check run
> after the rest of make has (possibly after a "build successful,
> continuing automatically to run tests" message so the impatient can
> interrupt it at that point), at the same time as the rest of make
> (increasing time-to-test, IMHO) or asynchronously after make has
> finished?

These are all good questions.  :-)

Off the top of my head without thinking about it more than ten seconds:
Since this is a low-cost low-effort thing, I think the timestamp should
just be updated when it's run, and just a single timestamp, no matter
how much fails.

So if you do a git pull, and that updates foo.el and bar.el, then "make"
would (as now) compile foo.el and bar.el, and Emacs would then notice
that foo.elc and bar.elc are newer than the previous timestamp, and then
run the foo-tests.el and bar-tests.elc files, and then update the
timestamp.

I think that might be unnoticeable enough that developers wouldn't be
disabling this when working?

Of course, if we rely on this too much, then there's the temptation to
never actually run a full "make check", but we have the CI for that, so
perhaps that's not too big a worry.

> Speaking of code changes, one thing I'm worried about is that one
> developer on, say, Windows does something, forgets to adjust the tests
> so there's a spurious failure on GNU/Linux, and then the next
> GNU/Linux developer to touch the file will see a test failure that has
> nothing to do with their changes. I don't have a good solution for
> that one.

Me neither.  But that's also the case today.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no