Re: [PATCH] tests/qtest/migration-test: Disable migration/multifd/tcp/plain/cancel

[Juan Quintela]

Daniel P. Berrangé <berrange@redhat.com> wrote:
> On Thu, Mar 02, 2023 at 05:22:11PM +0000, Peter Maydell wrote:
>> migration-test has been flaky for a long time, both in CI and
>> otherwise:
>> 
>> https://gitlab.com/qemu-project/qemu/-/jobs/3806090216
>> (a FreeBSD job)
>>   32/648 
>> ERROR:../tests/qtest/migration-helpers.c:205:wait_for_migration_status: 
>> assertion failed: (g_test_timer_elapsed() < MIGRATION_STATUS_WAIT_TIMEOUT) 
>> ERROR
>> 
>> on a local macos x86 box:
>> ▶  34/621 
>> ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed: 
>> assertion failed: (!g_str_equal(status, "failed")) ERROR
>>  34/621 qemu:qtest+qtest-i386 / qtest-i386/migration-test                    
>>      ERROR          168.12s   killed by signal 6 SIGABRT
>> ――――――――――――――――――――――――――――――――――――― ✀  
>> ―――――――――――――――――――――――――――――――――――――
>> stderr:
>> qemu-system-i386: Failed to peek at channel
>> query-migrate shows failed migration: Unable to write to socket: Broken pipe
>> **
>> ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed: 
>> assertion failed: (!g_str_equal(status, "failed"))
>> 
>> (test program exited with status code -6)
>> ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
>> 
>> ▶  37/621 
>> ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed: 
>> assertion failed: (!g_str_equal(status, "failed")) ERROR
>>  37/621 qemu:qtest+qtest-x86_64 / qtest-x86_64/migration-test                
>>      ERROR          174.37s   killed by signal 6 SIGABRT
>> ――――――――――――――――――――――――――――――――――――― ✀  
>> ―――――――――――――――――――――――――――――――――――――
>> stderr:
>> query-migrate shows failed migration: Unable to write to socket: Broken pipe
>> **
>> ERROR:../../tests/qtest/migration-helpers.c:151:migrate_query_not_failed: 
>> assertion failed: (!g_str_equal(status, "failed"))
>> 
>> (test program exited with status code -6)
>> 
>> In the cases where I've looked at the underlying log, this seems to
>> be in the migration/multifd/tcp/plain/cancel subtest.  Disable that
>> specific subtest by default until somebody can track down the
>> underlying cause. Enthusiasts can opt back in by setting
>> QEMU_TEST_FLAKY_TESTS=1 in their environment.
>
> No objection to disabling the test. Given the many multifd fixes we
> have seen, I fear that unlikely many of the flakey tests, this is
> not merely a test problem, but rather has a decent chance of being
> a real bug in migration code.

What is really weird with this failure is that:
- it only happens on non-x86
- on code that is not arch dependent
- on cancel, what we really do there is close fd's for the multifd
  channel threads to get out of the recv, i.e. again, nothing that
  should be arch dependent.

As said in the other email, I expect to get back access to ARM servers
next week.

Later, Juan.