bug#30526: java-testng: non-deterministic test failures

[Gábor Boskovits]

2018-02-25 12:16 GMT+01:00 Julien Lepiller <address@hidden>:

Le Mon, 19 Feb 2018 14:55:04 +0100,
Ricardo Wurmus <address@hidden> a écrit :

> Sometimes I get this error when building java-testng 6.14.2:
>
> --8<---------------cut here---------------start------------->8---
> ===============================================
> TestNG
> Total tests run: 1630, Failures: 1, Skips: 0
> ===============================================
>
> Failures in <suite> :TestNG, <test> :Regression2
> test.testng387.TestNG387.testInvocationCounterIsCorrectForMethodWithDataProvider()
> StackTrace:
>  java.lang.AssertionError: Arrays do not have the same size:3 != 4
>         at
> test.testng387.TestNG387.testInvocationCounterIsCorrectForMethodWithDataProvider(Unknown
> Source) ... Removed 27 stack frames
>
> phase `check' failed after 297.3 seconds
> --8<---------------cut here---------------end--------------->8---
>
>

Hi,

I've updated testng to the latest version (6.14.3) and it seems to
solve the issue. Can you confirm it?

 

Ok, I will check that today. 

It seems, that these are related to some race conditions in the parallel execution of tests. We also have verifyThatTestMethodsRunInParallelThreads failing sometimes. I will report back with results today.