Re: Display test suite

[Clément Pit--Claudel]

On 2016-07-07 12:29, Eli Zaretskii wrote:
>> From: Clément Pit--Claudel <address@hidden> Date: Thu, 7 Jul
>> 2016 11:47:54 -0400
>> 
>> Richard recently suggested giving Emacs the ability to take a
>> screenshot of itself. Could we imagine using this to run regression
>> tests of the display engine?
> 
> Maybe I'm missing something, but I don't see how.  A test suite
> needs to include the expected results for each test.  Even if you
> include an image file for each test that shows what should be
> displayed, AFAIU the image will be specific to the display
> resolution, number of colors, dimensions, etc., so inappropriate for
> automatic comparison with the screenshot on the target machine.  Am I
> right?

You're right; that's why I can only run the company-coq tests on my machine. 
Still, one might imagine setting up Emacs in a particular way to make them 
reproducible. Along these lines, I run the PNG through an optimizer and strip 
the metadata to make sure that I get the same output every time.

> Also, comparing images that could have a small number of pixels 
> different in some corner is error-prone.

There are very nice tools to do image diffs; they will highlight the offending 
pixels in a different color.

> IMO, a test suite for the display engine needs a way to describe 
> precisely what's on the screen, in some language, and then compare 
> those descriptions.

Yes, that would be nice (we have something like that for font-lock, right?)

>> Of course, it could be hard to get screeenshots to looks exactly
>> the same for everyone. Instead, we could have a makefile target
>> that builds a large collection of these screenshots, and another
>> one that rebuilds it and compares against a previously generated
>> one.
> 
> But then each time I start using a new machine, my reference point
> is reset to a newer date, and I lose the ability for comparing
> against older versions.

Absolutely. I think such tests would serve as a nice way to check your own 
commits (for before/after tests).

Also, maybe the variability is a good point: each people regularly run these 
tests, then changes that affect only one configuration would be noticed. If we 
have tests that produce the same output for every one, changes that affect only 
one particular set of configurations would not be noticed.

In any case, I've found the graphical tests that I do for company-coq very 
useful; whether they could generalize to all of Emacs is a non-trivial question.

Cheers,
Clément.

signature.asc
Description: OpenPGP digital signature