Re: [lmi] Headless GUI tests

[Vadim Zeitlin]

On Fri, 12 Nov 2021 22:17:59 +0000 Greg Chicares <gchicares@sbcglobal.net> 
wrote:

GC> In a freshly-generated 'bookworm' chroot, running lmi's automated
GC> GUI test under 'xvfb' works, on my own machine at least.

 So the solution is clearly to update to Bookworm :-/

GC> It's slower than running the GUI test directly:

 I think we had checked it before and the explanation was that Xvfb is just
much slower than the regular X server for many operations (although,
confusingly, it is also faster for a few of them). So it shouldn't be
surprising that the timings differ, as some things just take so much longer
when using Xvfb.

GC> However, in my old 'bullseye' chroot, the GUI test works
GC> only if I run it directly, without 'xvfb':
GC> 
GC> /opt/lmi/bin[0]$time /opt/lmi/src/lmi/gui_test.sh
GC> NOTE: starting the test suite
GC> SUCCESS: 21 tests successfully completed.
GC> NOTE: 4 tests were skipped
GC> /opt/lmi/src/lmi/gui_test.sh  21.36s user 8.39s system 70% cpu 42.401 total
GC> 
GC> ...but certain tests fail if I run it via 'xvfb-run':
GC> 
GC> /opt/lmi/bin[0]$time xvfb-run /opt/lmi/src/lmi/gui_test.sh
GC> NOTE: starting the test suite
GC> paste_census: ERROR (Assertion 
'(lmi::ssize(grid_window->GetSelectedRows())) == (1)' failed (expected 1 vs 
observed 0). [wx_test_paste_census.cpp : 269] )

 Does it always start with this test failure or do different tests fail
depending on the run/phase on the moon?

 I don't see anything wrong with this particular test, but we could change
it, e.g. to perform selection in the grid from the keyboard instead of
using the mouse, and it might be enough to make it work?

GC> pdf_census: ERROR (Assertion 'composite_pdf.exists()' failed. 
[wx_test_pdf_create.cpp : 160] )
GC> pdf_illustration: ERROR (Assertion failure: Expected edit cell dialog to 
test $1,000,000,000 scaling was not shown. [file 
/opt/lmi/src/lmi/wx_test_new.hpp, line 62, in do_new_illustration()].)
GC> validate_output_census: ERROR (Assertion failure: Expected case defaults 
dialog was not shown. [file /opt/lmi/src/lmi/wx_test_validate_output.cpp, line 
150, in init_test_census()].)
GC> validate_output_illustration: ERROR (Assertion failure: Expected 
illustration properties dialog was not shown. [file 
/opt/lmi/src/lmi/wx_test_new.hpp, line 62, in do_new_illustration()].)
GC> validate_output_mec: ERROR (Assertion failure: Expected new MEC parameters 
dialog was not shown. [file /opt/lmi/src/lmi/wx_test_validate_output.cpp, line 
573, in run()].)
GC> FAILURE: 6 out of 25 tests failed.
GC> NOTE: 4 tests were skipped
GC> X connection to :99 broken (explicit kill or server shutdown).
GC> X connection to :99 broken (explicit kill or server shutdown).
GC> xvfb-run /opt/lmi/src/lmi/gui_test.sh  29.26s user 31.13s system 0% cpu 
3:35:14.96 total
GC> 
GC> [the "total" time is correct: it ran that long, in a forgotten
GC> session, until I eventually noticed it and killed it]

 I think this happens because some top level window (a dialog which could
have been shown later than expected, maybe?) must have remained open,
preventing the program from exiting. It is a bit difficult to check this,
as you can't look on the Xvfb screen, of course. But I think I should make
this code more robust in any case, either by extending the destructor of
ensure_top_window_closed to close all top level windows and not just the
main one, or by just calling ExitMainLoop() instead of using this class,
which would seem to be much simpler and more robust and I can't remember
why I hadn't done it like this from the beginning.

GC> ...and the same tests fail if I emulate 'xvfb-run' manually,

 This is rather reassuring, there should be no reason for anything to
behave any differently.

GC> Thus, in the old 'bullseye' chroot, with 'xvfb', some tests fail,
GC> and the process has to be killed manually, which is less than ideal
GC> for an automated test.

 Yes, I understand. But even if it did finish, it still wouldn't be ideal
if the tests failed without any good reason.

 BTW, I haven't tried it, but I think using "timeout" command should work
even with the GUI tests (although I've only ever used it with command line
programs), so maybe you could use it for now.

GC> Vadim, do you have any idea how I can debug this further?

 To be honest, no. I would probably try adding diagnostic printfs to the
code, including wx, to understand whether we're getting the expected
messages to try to find out where do things go wrong exactly, but this
would be quite time-consuming and might still not yield anything useful at
the end. Rather distastefully, blindly changing things around (e.g.
changing the test to perform the selection differently, as suggested
above), might be a more productive strategy in this particular case.

 How critical is fixing this, i.e. how big of a constraint is having to run
the test in your Bookworm chroot for you?
VZ

pgp8_MnYL2R5U.pgp
Description: PGP signature