[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Rethinking remote testing

From: Jacob Bachmeyer
Subject: Rethinking remote testing
Date: Thu, 29 Nov 2018 18:19:39 -0600
User-agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv: Gecko/20090807 MultiZilla/ SeaMonkey/1.1.17 Mnenhy/

The recent and ongoing discussion over is_remote (and why is that the only is* predicate with an underscore in the name?) and its less-than-entirely-accurate implementation started me thinking about the topic. What would be a better way to handle remote testing?

We will need to preserve backwards compatibility for testsuites, or at least provide for a transition period to move to the new model. Ideally, the only changes affecting testsuites should be that existing workarounds become unnecessary going forward. At the very least, the GDB workaround Pedro Alves mentioned must not break the new implementation.

Test configuration (global init files) need not remain backwards compatible -- a site can update their init files when installing the upgraded DejaGnu. Ideally, "old model" configuration should be automatically migrated in memory upon load.

A rough sketch to start a discussion for a future solution:

Add an abstraction layer (that probably will support existing API calls -- I *think* DejaGnu already has this) that wraps a logical connection. Remote testing uses a "transfer" connection (ftp/kermit/tftp/sftp/scp/rcp/rsync/etc.) and a "control" connection (telnet/rsh/ssh/kermit/snmp/etc.). Local testing uses a "local-unix" or "local-posix" connection that provides both interfaces, but implements "transfer" as no-ops. Ideally, testsuites should not care whether they are running in local or remote mode.

We want to support Canadian cross environments with a separate test driver machine (so possibly four different platforms, but DejaGnu itself is relatively platform-independent, needing only a POSIX host) and any combination of build/host/target for toolchain testing. Note that multiarch machines are likely to be the future: AMD64 already does this today, and RISC-V systems could have all of RV32/RV64/RV128 on the same machine, all with varying extensions either by configuration or heterogeneous hardware. Cross toolchains are likely to increase in importance. So we have toolchain "build"/"host"/"target" and a separate DejaGnu "driver" in the picture, all of which can be same or different, so a testsuite could use up to three remote connections.

This leads to a few scenarios off the top of my head that DejaGnu needs to support:

1. Toolchain: compilers. These (notably GCC but possibly others as well) can need the full complexity of the system, with possible need to build the tested program on one remote system, transfer the executables to another remote system, use the executables on "host" to produce test programs, transfer the test programs to a third remote system, and run the test programs on "target". The simplifying factor here is than compilers are batch programs, so interaction is limited.

2. Toolchain: debuggers. These (probably only GDB, but possibly others for specific languages) can be as complex as compilers, but are also interactive programs for which the tests can depend on a compiler to build test programs on "host" for "target". The debugger itself also runs on "host" (possibly a different "host" from the associated compiler!) and connects (by some means independent of DejaGnu) to "target" for debugging the test programs.

3. Unit testing. These are probably less complex, since there is only a need to build (possibly on a remote "build") the unit test programs, transfer them to the "host", run them on the "host", and collect the results. Small complexities are introduced when gcov(1) or similar tools are used to validate unit test coverage.

4. General system testing. These are again similar to unit testing: programs are built on "build", transferred, and run on "host". These are to unit tests as debuggers are to compilers: general system testing is likely to be "interactive" rather than specific "run this program".

Debuggers are particularly interesting, since they potentially introduce fifth and sixth(!) systems into the picture: the "build" and "host" systems for the associated compiler, which could be different from the "build" and "host" systems for the debugger. This further expands if there are multiple associated compilers. This suggests that we should split build/host/target into a hierarchical structure, with plain build/host/target as defaults for the "main tool" being tested, but allowing "subtools" (like the compiler in the case of a debugger) to have their own build/host/target sets. This could be particularly relevant for multi-arch systems, where a compiler notionally built for one target may well be capable of producing output either directly for or that is compatible with another target. (I expect RISC-V to be able to make extensive use of this eventually.)

The possibility of needing remote builds raises a question of scope: on one hand, it is easy to say that building tested programs is out-of-scope for DejaGnu, but on the other hand, DejaGnu already needs most of the relevant functionality in order to support remote testing of compilers. The incremental code is small: a routine that uses the already-needed remote connection infrastructure to transfer the sources to a build machine, run "configure" and "make", and collect the output binaries.

The complexity associated with these changes is probably unmanageable with DejaGnu's current design, which is why I advocate for moving the DejaGnu core and core libraries out of the global Tcl namespace. (Doing that (backwards compatibly) is on my local TODO list, by the way.)

Eventually, (far future) DejaGnu could offer a "strict" mode, where test scripts are run in safe interpreters with access to only the DejaGnu API. If the API is properly designed, a testsuite that runs in "strict" mode should be independent of local or remote configuration. Depending on the details of the GDB testsuite, "strict" mode may also require some notion of "test groups" -- some test scripts being applicable only on certain "boards".

Lastly, it unlikely that this discussion would even be possible if the existing DejaGnu, with all its limitations, had not been written. Thanks to Rob and Cygnus for laying that groundwork.

-- Jacob

reply via email to

[Prev in Thread] Current Thread [Next in Thread]