Re: [PATCH] tests/qtest/migration-test: Disable migration/multifd/tcp/plain/cancel

[Philippe Mathieu-Daudé]

On 2/3/23 18:37, Dr. David Alan Gilbert wrote:
> * Peter Maydell (peter.maydell@linaro.org) wrote:
> > migration-test has been flaky for a long time, both in CI and
> > otherwise:
> >
> > https://gitlab.com/qemu-project/qemu/-/jobs/3806090216
> > (a FreeBSD job)
> >    32/648 ERROR:../tests/qtest/migration-helpers.c:205:wait_for_migration_status: 
> > assertion failed: (g_test_timer_elapsed() < MIGRATION_STATUS_WAIT_TIMEOUT) ERROR
> >
> > on a local macos x86 box:
> > ▶  34/621 ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed: 
> > assertion failed: (!g_str_equal(status, "failed")) ERROR
> >   34/621 qemu:qtest+qtest-i386 / qtest-i386/migration-test                      
> >    ERROR          168.12s   killed by signal 6 SIGABRT
> > ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> > stderr:
> > qemu-system-i386: Failed to peek at channel
> > query-migrate shows failed migration: Unable to write to socket: Broken pipe
> > **
> > ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed: assertion 
> > failed: (!g_str_equal(status, "failed"))
> >
> > (test program exited with status code -6)
> > ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
> >
> > ▶  37/621 ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed: 
> > assertion failed: (!g_str_equal(status, "failed")) ERROR
> >   37/621 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test                  
> >    ERROR          174.37s   killed by signal 6 SIGABRT
> > ――――――――――――――――――――――――――――――――――――― ✀  ―――――――――――――――――――――――――――――――――――――
> > stderr:
> > query-migrate shows failed migration: Unable to write to socket: Broken pipe
> > **
> > ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed: assertion 
> > failed: (!g_str_equal(status, "failed"))
> >
> > (test program exited with status code -6)
> >
> > In the cases where I've looked at the underlying log, this seems to
> > be in the migration/multifd/tcp/plain/cancel subtest.  Disable that
> > specific subtest by default until somebody can track down the
> > underlying cause. Enthusiasts can opt back in by setting
> > QEMU_TEST_FLAKY_TESTS=1 in their environment.
> >
> > We might need to disable more parts of this test if this isn't
> > sufficient to fix the flakiness.
> >
> > Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
> > ---
> > This is a slightly more targeted variation on my original
> > modest proposal.
> > ---
> >   tests/qtest/migration-test.c | 10 ++++++++--
> >   1 file changed, 8 insertions(+), 2 deletions(-)
> >
> > diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
> > index 109bc8e7b13..d4ab3934ed2 100644
> > --- a/tests/qtest/migration-test.c
> > +++ b/tests/qtest/migration-test.c
> > @@ -2572,8 +2572,14 @@ int main(int argc, char **argv)
> >       qtest_add_func("/migration/auto_converge", test_migrate_auto_converge);
> >       qtest_add_func("/migration/multifd/tcp/plain/none",
> >                      test_multifd_tcp_none);
> > -    qtest_add_func("/migration/multifd/tcp/plain/cancel",
> > -                   test_multifd_tcp_cancel);
> > +    /*
> > +     * This test is flaky and sometimes fails in CI and otherwise:
> > +     * don't run unless user opts in via environment variable.
> > +     */
> > +    if (getenv("QEMU_TEST_FLAKY_TESTS")) {
> > +        qtest_add_func("/migration/multifd/tcp/plain/cancel",
> > +                       test_multifd_tcp_cancel);
> > +    }
>
> OK, that seems reasonably selective.
>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
>
> (Cancel tests are always a pain; they can be racy with the test
> completing before you fire the cancel; or 'cancel' itself
> can hit lots of races inside the migration code if it's not written
> carefully enough to expect a cancel).

Set 'QEMU_TEST_FLAKY_TESTS' in your gitlab namespace:
https://docs.gitlab.com/ee/ci/variables/#for-a-project

(Also, set it in your local env or in the script you run
to automate your testing).