[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PATCH 04/16] Acceptance Tests: keep track and disable tests with known
From: |
Cleber Rosa |
Subject: |
[PATCH 04/16] Acceptance Tests: keep track and disable tests with known issues |
Date: |
Fri, 24 Sep 2021 14:54:54 -0400 |
This introduces a convention, under which tests with known open
issues, will be automatically excluded from execution via "make
check-acceptance". It should by itself improve the reliability
of test jobs, even those with rare failures.
Those tests can still be run with:
./tests/venv/bin/avocado run $TEST_ID
With the test identification are obtainable via:
./tests/venv/bin/avocado list tests/acceptance
And for tests with known issues:
./tests/venv/bin/avocado list -t issue tests/acceptance
A list of tests that are enabled (that is, match the currently
configured targets, and do not have known issues) can be obtained
with:
make list-acceptance
As a future improvement, a tool can be added to sync the "issue" tags
with issues that end up being resolved closed.
Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
docs/devel/testing.rst | 28 ++++++++++++++++++++++++++
tests/Makefile.include | 3 ++-
tests/acceptance/boot_linux_console.py | 3 +++
tests/acceptance/machine_rx_gdbsim.py | 3 +++
4 files changed, 36 insertions(+), 1 deletion(-)
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 2818c58ff8..f93a9bed3f 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -1147,6 +1147,34 @@ variable as shown on the code snippet below to skip the
test:
def test(self):
do_something()
+Disabling tests
+---------------
+
+If you want to have a test temporarily disabled (from ``make
+check-acceptance``), you can add an Avocado ``issue`` tag to the test.
Example::
+
+ class Feature(Test):
+
+ def test(self):
+ """
+ :avocado: tags=issue
+ """
+ do_something()
+
+Rather than simply adding an ``issue`` tag, it may be helpful to point
+to an actual issue on the project's issue tracker::
+
+ class Feature(Test):
+
+ def test(self):
+ """
+ Reason for disabling this test is documented and tracked at
+ https://gitlab.com/qemu-project/qemu/-/issues/9999
+
+ :avocado: tags=issue:9999
+ """
+ do_something()
+
Uninstalling Avocado
--------------------
diff --git a/tests/Makefile.include b/tests/Makefile.include
index d1f90572a7..eb3e5d2aa6 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -94,7 +94,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
# information please refer to "avocado --help".
AVOCADO_SHOW=app
AVOCADO_RUNNER=runner
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
+comma := ,
+AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=-issue$(comma)arch:%,
$(filter %-softmmu,$(TARGETS)))
$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
$(call quiet-command, \
diff --git a/tests/acceptance/boot_linux_console.py
b/tests/acceptance/boot_linux_console.py
index 0a49c0e276..4c824bc161 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -485,6 +485,9 @@ def test_arm_raspi2_initrd(self):
"""
:avocado: tags=arch:arm
:avocado: tags=machine:raspi2b
+
+ # https://gitlab.com/qemu-project/qemu/-/issues/636
+ :avocado: tags=issue:636
"""
deb_url = ('http://archive.raspberrypi.org/debian/'
'pool/main/r/raspberrypi-firmware/'
diff --git a/tests/acceptance/machine_rx_gdbsim.py
b/tests/acceptance/machine_rx_gdbsim.py
index 32b737b6d8..62893a9989 100644
--- a/tests/acceptance/machine_rx_gdbsim.py
+++ b/tests/acceptance/machine_rx_gdbsim.py
@@ -54,6 +54,9 @@ def test_linux_sash(self):
:avocado: tags=arch:rx
:avocado: tags=machine:gdbsim-r5f562n7
:avocado: tags=endian:little
+
+ # https://gitlab.com/qemu-project/qemu/-/issues/507
+ :avocado: tags=issue:507
"""
dtb_url = ('https://acc.dl.osdn.jp/users/23/23887/rx-virt.dtb')
dtb_hash = '7b4e4e2c71905da44e86ce47adee2210b026ac18'
--
2.31.1
- Re: [PATCH 05/16] Acceptance Tests: add standard clean up at test tearDown(), (continued)
- [PATCH 06/16] Acceptance Tests: use extract from package from avocado.utils, Cleber Rosa, 2021/09/24
- [PATCH 07/16] Acceptance Tests: workaround expired mipsdistros.mips.com HTTPS cert, Cleber Rosa, 2021/09/24
- [PATCH 08/16] acceptance/tests/vnc.py: use explicit syntax for enabling passwords, Cleber Rosa, 2021/09/24
- [PATCH 09/16] tests/acceptance/boot_xen.py: merge base classes, Cleber Rosa, 2021/09/24
- [PATCH 03/16] Acceptance Tests: add mechanism for listing tests, Cleber Rosa, 2021/09/24
- [PATCH 10/16] tests/acceptance/boot_xen.py: unify tags, Cleber Rosa, 2021/09/24
- [PATCH 02/16] Acceptance Tests: improve check-acceptance description, Cleber Rosa, 2021/09/24
- [PATCH 04/16] Acceptance Tests: keep track and disable tests with known issues,
Cleber Rosa <=
- [PATCH 11/16] tests/acceptance/boot_xen.py: fetch kernel during test setUp(), Cleber Rosa, 2021/09/24
- [PATCH 12/16] tests/acceptance/boot_xen.py: removed unused import, Cleber Rosa, 2021/09/24
- [PATCH 13/16] tests/acceptance/boot_xen.py: use class attribute, Cleber Rosa, 2021/09/24
- [PATCH 14/16] tests/acceptance/ppc_prep_40p.py: NetBSD 7.1.2 location update, Cleber Rosa, 2021/09/24
- [PATCH 15/16] tests/acceptance/ppc_prep_40p.py: clean up unused import, Cleber Rosa, 2021/09/24