dejagnu
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Enhancement request: `--dry-run` flag for `runtest`


From: Jacob Bachmeyer
Subject: Re: Enhancement request: `--dry-run` flag for `runtest`
Date: Fri, 06 May 2022 16:51:43 -0500
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.22) Gecko/20090807 MultiZilla/1.8.3.4e SeaMonkey/1.1.17 Mnenhy/0.7.6.0

Eric Gallager wrote:
Many command-line utilities have a "--dry-run" flag to print what
would be done by running them without actually performing any changes.
I was thinking one would be particularly useful for running the GCC
testsuite (to put in RUNTESTFLAGS), since it is quite large and
running it can take quite a long time, yet sometimes there are reasons
to run it besides actually running it (e.g., getting a list of tests
run). For example, https://gcc.gnu.org/install/test.html says: "To get
a list of the possible *.exp files, pipe the output of ‘make check’
into a file and look at the ‘Running … .exp’ lines," which is
something that I thought being able to do "RUNTEST_FLAGS=--dry-run"
would be useful for.

The main problem with this suggestion is technical: the DejaGnu framework does not know what tests are in the testsuite (adding means to declare this is one of my eventual plans, if I can find a way to make it fit DejaGnu's architecture). The problem is that it is possible for a test to silently misfire and produce no result at all, depending on how the test scripts are written, which comes back to the problem that makes '--dry-run' difficult for DejaGnu: we could list the test scripts that would be run without actually running them, but without support in the testsuite there is no way to know what tests each script would do, and there is infrastructure in DejaGnu to support test scripts that use some (potentially variable) set of external files. Without actually running the tests, the framework knows nothing about those files.

Another possibility would be to replace host_execute, target_compile, and others with stubs when in dry-run mode and ignore results (replacing the procedures that actually run tests _should_ cause numerous failures in most testsuites); this would allow to evaluate the test scripts, but may cause confusion, especially in more advanced testsuites, like GCC.

For your use case, is this really needed? DejaGnu has logic that is (or should be) very similar to `find testsuite/ -type d -name testsuite -prune -o -type d -name config -prune -o -type d -name lib -prune -o -name '*.exp' -print` for locating test scripts to run, except that DejaGnu additionally narrows the list to files appropriate for the selected tool to be tested and also excludes directories associated with Git, Subversion, CVS, RCS, and SCCS from the search. In a large package like GCC that has many testsuites, you could first find the testsuites with a simple `find . -type d -name testsuite -print`.

DejaGnu is beginning to accumulate a selection of auxiliary tools, and finding the scripts that runtest could possibly use might be a helpful feature there. Perhaps `dejagnu list testsuites` to wrap the second find command I mentioned and `dejagnu list test scripts` for the more detailed search of one specific testsuite (always or usually ./testsuite)?

Let me know if you need me to find specific
examples of software that already has the kind of "--dry-run" flag I'm
thinking of.
I suspect that GNU make is a good example.


-- Jacob





reply via email to

[Prev in Thread] Current Thread [Next in Thread]