Re: [PATCH 2/2] tests/avocado/boot_linux.py: re-enable test-case for ppc64

[Cédric Le Goater]

Hello,

[ Copying qemu-ppc@ and Daniel ]

On 3/28/23 13:24, Kautuk Consul wrote:
> On 2023-03-27 17:07:30, Alex Bennée wrote:
> > Kautuk Consul <kconsul@linux.vnet.ibm.com> writes:
> >
> > > Fixes c0c8687ef0("tests/avocado: disable BootLinuxPPC64 test in CI").
> > >
> > > Commit c0c8687ef0fd990db8db1655a8a6c5a5e35dd4bb disabled the test-case
> > > for PPC64. On investigation, this turns out to be an issue with the
> > > time taken for downloading the Fedora 31 qcow2 image being included
> > > within the test-case timeout.
> > > Re-enable this test-case by setting the timeout to 360 seconds just
> > > before launching the downloaded VM image.
> > >
> > > Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
> > > Reported-by: Alex Bennée <alex.bennee@linaro.org>
> > > Tested-by: Hariharan T S hariharan.ts@linux.vnet.ibm.com
> >
> > It doesn't really address the principle problem that the
> > boot_linux.py:BootLinuxPPC64.test_pseries_tcg is super heavyweight for
> > only 2% extra coverage of the executed lines.
> By re-enabling this test-case we will ensure that PPC64 part of qemu
> works okay in terms of basic linux boot. Without this we will have
> a regression in the sense that there won't be any way to test out
> basic linux boot for PPC64.

There are ways and pseries is not only PPC64 machine. There is more
to it. See :

  https://github.com/legoater/qemu-ppc-boot/tree/main/buildroot
  https://github.com/legoater/buildroot/tree/qemu-ppc/board/qemu

QEMU PPC maintainers have external tools for regressions which are
run regularly, at least before sending a PR for upstream.

Thanks,

C.

> > What we really need is a script so we can compare the output between the
> > two jsons:
> >
> >    gcovr --json --exclude-unreachable-branches --print-summary -o coverage.json 
> > --root ../../ . *.p
> >
> > because I suspect we could make up that missing few % noodling the
> > baseline test a bit more.
> Can you tell me how you check code coverage with and without this
> test-case ? I am kind of new to qemu so it would be nice to know how you
> do this. And I am trying to increase the code coverage by improving
> the baseline test by including more devices in the qemu-system-ppc64
> command line so I would appreciate any tips on how to do that also.
> > > ---
> > >   tests/avocado/boot_linux.py | 6 +++++-
> > >   1 file changed, 5 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/tests/avocado/boot_linux.py b/tests/avocado/boot_linux.py
> > > index be30dcbd58..c3869a987c 100644
> > > --- a/tests/avocado/boot_linux.py
> > > +++ b/tests/avocado/boot_linux.py
> > > @@ -91,9 +91,9 @@ class BootLinuxPPC64(LinuxTest):
> > >       :avocado: tags=arch:ppc64
> > >       """
> > >   
> > > +    # timeout for downloading new VM image.
> > >       timeout = 360
> > >   
> > > -    @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
> > >       def test_pseries_tcg(self):
> > >           """
> > >           :avocado: tags=machine:pseries
> > > @@ -101,6 +101,10 @@ def test_pseries_tcg(self):
> > >           """
> > >           self.require_accelerator("tcg")
> > >           self.vm.add_args("-accel", "tcg")
> > > +
> > > +        # timeout for actual Linux PPC boot test
> > > +        self.timeout = 360
> > > +
> > >           self.launch_and_wait(set_up_ssh_connection=False)
> >
> >
> > --
> > Alex Bennée
> > Virtualisation Tech Lead @ Linaro