|
| From: | Krishan Gopal |
| Subject: | Re: [PATCH] Skip tests if tool/dependency is not installed |
| Date: | Wed, 14 Feb 2024 11:13:43 +0530 |
| User-agent: | Mozilla Thunderbird |
Hi Glenn,
Hi Krishan, On Tue, 30 Jan 2024 04:18:13 -0600 Krishan Gopal Saraswat <krishang@linux.ibm.com> wrote:Currently many tests ends up in an ERROR state due to the environment not having relevant tools, majorly being due to different file systems. For example before this patch the ntfs test fails with the following error: mkfs.ntfs not installed; cannot test ntfs. ERROR ntfs_test (exit status: 99) This was caused due to it returning 99 exit status, which represents ERROR. If a particular tool is not installed/ dependecies not met, it should go to SKIP state.This is working as intended. An exit code of 99 represents a "hard error"[1], which is not a normal error and indicates a failure that results from an improperly run test. This is to distinguish from errors that we actually care about resulting from properly run tests. A skipped test, exit code 77, are for tests that should not actually be run, as in the case of tests that do not apply to the target being used. This matters because those running tests should be notified that they are not properly running all the tests (usually due to lack of dependencies). Skipping those failing tests obscures this fact. The point of the test suite is to have all the tests run, not to only run the successful ones and ignore the rest. What problem does this patch actually solve?
Thanks for the review,
My view of SKIP/ERROR is the following.
Considering the following two conditions for SKIP: What ever not supported on a particular platform [powerPC]. For example NTFS file system is for Windows and not supported on Linux - in this case expecting a SKIP instead of ERROR Considering the following for ERROR: Expected tool is available but failed due to some error. This patch will help in getting the correct output for a test execution that is not supported.Thanks,
Krishan
Glenn [1] https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.htmlAfter this patch, ntfs test skips if the tool is not available, with the following message: mkfs.ntfs not installed; cannot test ntfs. SKIP ntfs_test (exit status: 77) Many such test cases had similar failures and have been modified to SKIP state. The following test cases also had similar failures and have been modified from ERROR to SKIP state: hfsplus_test, ntfs_test, reiserfs_test, f2fs_test, nilfs2_test, romfs_test, exfat_test, udf_test, hfs_test, jfs_test, btrfs_test, zfs_test, xfs_test Signed-off-by: Krishan Gopal Saraswat <krishang@linux.ibm.com> --- btrfs_test.in | 2 +- exfat_test.in | 2 +- f2fs_test.in | 2 +- hfs_test.in | 4 ++-- hfsplus_test.in | 2 +- jfs_test.in | 2 +- nilfs2_test.in | 2 +- ntfs_test.in | 4 ++-- reiserfs_test.in | 2 +- romfs_test.in | 2 +- udf_test.in | 2 +- xfs_test.in | 2 +- zfs_test.in | 2 +- 13 files changed, 15 insertions(+), 15 deletions(-) diff --git a/btrfs_test.in b/btrfs_test.in index 0d098c9..a07d2e5 100644 --- a/btrfs_test.in +++ b/btrfs_test.in @@ -12,7 +12,7 @@ fi if ! which mkfs.btrfs >/dev/null 2>&1; then echo "mkfs.btrfs not installed; cannot test btrfs." - exit 99 + exit 77 fi "@builddir@/grub-fs-tester" btrfs diff --git a/exfat_test.in b/exfat_test.in index 7939f25..7acde19 100644 --- a/exfat_test.in +++ b/exfat_test.in @@ -12,7 +12,7 @@ fi if ! which mkfs.exfat >/dev/null 2>&1; then echo "mkfs.exfat not installed; cannot test exFAT." - exit 99 + exit 77 fi "@builddir@/grub-fs-tester" exfat diff --git a/f2fs_test.in b/f2fs_test.in index 85f8cc8..a020a0f 100644 --- a/f2fs_test.in +++ b/f2fs_test.in @@ -12,7 +12,7 @@ fi if ! which mkfs.f2fs >/dev/null 2>&1; then echo "mkfs.f2fs not installed; cannot test f2fs." - exit 99 + exit 77 fi diff --git a/hfs_test.in b/hfs_test.in index 960f1cb..077683c 100644 --- a/hfs_test.in +++ b/hfs_test.in @@ -12,12 +12,12 @@ fi if ! which mkfs.hfs >/dev/null 2>&1; then echo "mkfs.hfs not installed; cannot test HFS." - exit 99 + exit 77 fi if ! grep -q mac_roman /proc/modules && ! modprobe mac_roman; then echo "no mac-roman support; cannot test HFS." - exit 99 + exit 77 fi "@builddir@/grub-fs-tester" hfs diff --git a/hfsplus_test.in b/hfsplus_test.in index f727cf0..cb36a36 100644 --- a/hfsplus_test.in +++ b/hfsplus_test.in @@ -12,7 +12,7 @@ fi if ! which mkfs.hfsplus >/dev/null 2>&1; then echo "mkfs.hfsplus not installed; cannot test hfsplus." - exit 99 + exit 77 fi "@builddir@/grub-fs-tester" hfsplus diff --git a/jfs_test.in b/jfs_test.in index d13780e..86f9ebe 100644 --- a/jfs_test.in +++ b/jfs_test.in @@ -12,7 +12,7 @@ fi if ! which mkfs.jfs >/dev/null 2>&1; then echo "mkfs.jfs not installed; cannot test JFS." - exit 99 + exit 77 fi "@builddir@/grub-fs-tester" jfs diff --git a/nilfs2_test.in b/nilfs2_test.in index 8cc9375..719972f 100644 --- a/nilfs2_test.in +++ b/nilfs2_test.in @@ -12,7 +12,7 @@ fi if ! which mkfs.nilfs2 >/dev/null 2>&1; then echo "mkfs.nilfs2 not installed; cannot test nilfs2." - exit 99 + exit 77 fi "@builddir@/grub-fs-tester" nilfs2 diff --git a/ntfs_test.in b/ntfs_test.in index c2b08d2..da73c59 100644 --- a/ntfs_test.in +++ b/ntfs_test.in @@ -12,12 +12,12 @@ fi if ! which mkfs.ntfs >/dev/null 2>&1; then echo "mkfs.ntfs not installed; cannot test ntfs." - exit 99 + exit 77 fi if ! which setfattr >/dev/null 2>&1; then echo "setfattr not installed; cannot test ntfs." - exit 99 + exit 77 fi "@builddir@/grub-fs-tester" ntfs diff --git a/reiserfs_test.in b/reiserfs_test.in index 37226c0..36e34c3 100644 --- a/reiserfs_test.in +++ b/reiserfs_test.in @@ -12,7 +12,7 @@ fi if ! which mkfs.reiserfs >/dev/null 2>&1; then echo "mkfs.reiserfs not installed; cannot test reiserfs." - exit 99 + exit 77 fi "@builddir@/grub-fs-tester" reiserfs diff --git a/romfs_test.in b/romfs_test.in index f968e9b..98bb50c 100644 --- a/romfs_test.in +++ b/romfs_test.in @@ -4,7 +4,7 @@ set -e if ! which genromfs >/dev/null 2>&1; then echo "genromfs not installed; cannot test romfs." - exit 99 + exit 77 fi "@builddir@/grub-fs-tester" romfs diff --git a/udf_test.in b/udf_test.in index 302b28a..8968fb1 100644 --- a/udf_test.in +++ b/udf_test.in @@ -12,7 +12,7 @@ fi if ! which mkudffs >/dev/null 2>&1; then echo "mkudffs not installed; cannot test UDF." - exit 99 + exit 77 fi "@builddir@/grub-fs-tester" udf diff --git a/xfs_test.in b/xfs_test.in index 5e029c1..8a648aa 100644 --- a/xfs_test.in +++ b/xfs_test.in @@ -12,7 +12,7 @@ fi if ! which mkfs.xfs >/dev/null 2>&1; then echo "mkfs.xfs not installed; cannot test xfs." - exit 99 + exit 77 fi diff --git a/zfs_test.in b/zfs_test.in index 58cc25b..5d0f07d 100644 --- a/zfs_test.in +++ b/zfs_test.in @@ -12,7 +12,7 @@ fi if ! which zpool >/dev/null 2>&1; then echo "zpool not installed; cannot test zfs." - exit 99 + exit 77 fi "@builddir@/grub-fs-tester" zfs
| [Prev in Thread] | Current Thread | [Next in Thread] |