Re: [PATCH 3/3] tests/avocado: Enable reverse_debugging.py tests in gitlab CI

[Daniel P . Berrangé]

On Tue, Nov 21, 2023 at 09:56:24AM +0100, Thomas Huth wrote:
> On 17/11/2023 08.35, Nicholas Piggin wrote:
> > On Fri Nov 17, 2023 at 4:11 AM AEST, Thomas Huth wrote:
> > > On 16/11/2023 12.53, Nicholas Piggin wrote:
> > > > Let's try enable reverse_debugging.py in gitlab CI.
> > > > 
> > > > Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> > > > ---
> > > > Maybe we could try this again at some point? The bug might have been
> > > > noticed sooner.
> > > > 
> > > > They only take a couple of seconds to run so should not take too much
> > > > overhead. But my gitlab CI pipeline doesn't run the avocado tests for
> > > > some reason, so I can't see if it's still causing problems.
> > > > 
> > > > Thanks,
> > > > Nick
> > > > ---
> > > > 
> > > >    tests/avocado/reverse_debugging.py | 7 -------
> > > >    1 file changed, 7 deletions(-)
> > > 
> > > FYI, I gave it a try, and it survived my CI run:
> > > 
> > >    https://gitlab.com/thuth/qemu/-/jobs/5552213972#L403
> > > 
> > > So I went ahead and put it (together with the first patch) in my current
> > > pull request, let's see how it goes...
> > 
> > Great, thank you.
> 
> ... and here it's failing again (current master branch):
> 
> https://gitlab.com/thuth/qemu/-/jobs/5582657378#L404
> 
> According to the debug.log in the artifacts, it's failing here:
> 
> 08:28:32 DEBUG| [    0.230392217,5] OPAL v7.0 starting...
> 
> 08:28:32 DEBUG| [    0.230674939,7] initial console log level: memory 7, 
> driver 5
> 
> 08:28:32 DEBUG| [    0.231048494,6] CPU: P9 generation processor (max 4 
> threads/core)
> 
> 08:28:32 DEBUG| [
> 08:28:32 DEBUG| [    0.231412547,7] CPU: Boot CPU PIR is 0x0000 PVR is 
> 0x004e1202
> 
> 08:28:32 DEBUG| [
> 08:28:32 ERROR|
> 08:28:32 ERROR| Reproduced traceback from: 
> /builds/thuth/qemu/build/pyvenv/lib64/python3.8/site-packages/avocado/core/test.py:770
> 08:28:32 ERROR| Traceback (most recent call last):
> 08:28:32 ERROR|   File 
> "/builds/thuth/qemu/build/tests/avocado/reverse_debugging.py", line 262, in 
> test_ppc64_powernv
> 08:28:32 ERROR|     self.reverse_debugging()
> 08:28:32 ERROR|   File 
> "/builds/thuth/qemu/build/tests/avocado/reverse_debugging.py", line 178, in 
> reverse_debugging
> 08:28:32 ERROR|     g.cmd(b'c')
> 08:28:32 ERROR|   File 
> "/builds/thuth/qemu/build/pyvenv/lib64/python3.8/site-packages/avocado/utils/gdb.py",
>  line 783, in cmd
> 08:28:32 ERROR|     response_payload = self.decode(result)
> 08:28:32 ERROR|   File 
> "/builds/thuth/qemu/build/pyvenv/lib64/python3.8/site-packages/avocado/utils/gdb.py",
>  line 738, in decode
> 08:28:32 ERROR|     raise InvalidPacketError
> 08:28:32 ERROR| avocado.utils.gdb.InvalidPacketError
> 08:28:32 ERROR|
> 08:28:32 DEBUG| Local variables:
> 08:28:32 DEBUG|  -> self <class 'reverse_debugging.ReverseDebugging_ppc64'>: 
> 79-tests/avocado/reverse_debugging.py:ReverseDebugging_ppc64.test_ppc64_powernv
> 08:28:32 DEBUG| Shutting down VM appliance; timeout=30
> 08:28:32 DEBUG| Attempting graceful termination
> 08:28:32 DEBUG| Closing console socket
> 08:28:32 DEBUG| Politely asking QEMU to terminate
> 
> So unless someone has a clue how to fix that, I guess it's
> likely best to revert this enablement patch again...

A little further in the log we see

08:28:32 DEBUG| Politely asking QEMU to terminate
08:28:32 DEBUG| --> {
  "execute": "quit"
}
08:28:32 DEBUG| <-- {
  "timestamp": {
    "seconds": 1700555312,
    "microseconds": 86122
  },
  "event": "RESUME"
}
08:28:32 ERROR| Task.Reader: BrokenPipeError: [Errno 32] Broken pipe



With seeing a bad packet from GDB and seeing Broken pipe from QMP,
my impression is that the QEMU process is no longer present, most
likely it has SEGV'd I reckon.

IOW, I think we might well have a genuine bug here, not merely an
unreliable test suite.

None the less, unless someone can guess what the problem is, we'll
need to disable the test to get reliable CI.

A bug should be opened though with the CI logs.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|