Re: Enhancement request: `--dry-run` flag for `runtest`

[Jacob Bachmeyer]

Eric Gallager wrote:
> Many command-line utilities have a "--dry-run" flag to print what
> would be done by running them without actually performing any changes.
> I was thinking one would be particularly useful for running the GCC
> testsuite (to put in RUNTESTFLAGS), since it is quite large and
> running it can take quite a long time, yet sometimes there are reasons
> to run it besides actually running it (e.g., getting a list of tests
> run). For example, https://gcc.gnu.org/install/test.html says: "To get
> a list of the possible *.exp files, pipe the output of ‘make check’
> into a file and look at the ‘Running … .exp’ lines," which is
> something that I thought being able to do "RUNTEST_FLAGS=--dry-run"
> would be useful for.
The main problem with this suggestion is technical:  the DejaGnu 
framework does not know what tests are in the testsuite (adding means to 
declare this is one of my eventual plans, if I can find a way to make it 
fit DejaGnu's architecture).  The problem is that it is possible for a 
test to silently misfire and produce no result at all, depending on how 
the test scripts are written, which comes back to the problem that makes 
'--dry-run' difficult for DejaGnu:  we could list the test scripts that 
would be run without actually running them, but without support in the 
testsuite there is no way to know what tests each script would do, and 
there is infrastructure in DejaGnu to support test scripts that use some 
(potentially variable) set of external files.  Without actually running 
the tests, the framework knows nothing about those files.
Another possibility would be to replace host_execute, target_compile, 
and others with stubs when in dry-run mode and ignore results (replacing 
the procedures that actually run tests _should_ cause numerous failures 
in most testsuites); this would allow to evaluate the test scripts, but 
may cause confusion, especially in more advanced testsuites, like GCC.
For your use case, is this really needed?  DejaGnu has logic that is (or 
should be) very similar to `find testsuite/ -type d -name testsuite 
-prune -o -type d -name config -prune -o -type d -name lib -prune -o 
-name '*.exp' -print` for locating test scripts to run, except that 
DejaGnu additionally narrows the list to files appropriate for the 
selected tool to be tested and also excludes directories associated with 
Git, Subversion, CVS, RCS, and SCCS from the search.  In a large package 
like GCC that has many testsuites, you could first find the testsuites 
with a simple `find . -type d -name testsuite -print`.
DejaGnu is beginning to accumulate a selection of auxiliary tools, and 
finding the scripts that runtest could possibly use might be a helpful 
feature there.  Perhaps `dejagnu list testsuites` to wrap the second 
find command I mentioned and `dejagnu list test scripts` for the more 
detailed search of one specific testsuite (always or usually ./testsuite)?
> Let me know if you need me to find specific
> examples of software that already has the kind of "--dry-run" flag I'm
> thinking of.
I suspect that GNU make is a good example.


-- Jacob