[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
01/01: announcements: Add "GuixSD system tests" post.
From: |
Ludovic Courtčs |
Subject: |
01/01: announcements: Add "GuixSD system tests" post. |
Date: |
Tue, 28 Jun 2016 11:56:44 +0000 (UTC) |
civodul pushed a commit to branch master
in repository maintenance.
commit 51a0c31e5f47b0b9f7d80fbdffbfdf132b663904
Author: Ludovic Courtès <address@hidden>
Date: Tue Jun 28 13:56:12 2016 +0200
announcements: Add "GuixSD system tests" post.
---
doc/announcements/savannah/testing.txt | 114 ++++++++++++++++++++++++++++++++
1 file changed, 114 insertions(+)
diff --git a/doc/announcements/savannah/testing.txt
b/doc/announcements/savannah/testing.txt
new file mode 100644
index 0000000..f0ce555
--- /dev/null
+++ b/doc/announcements/savannah/testing.txt
@@ -0,0 +1,114 @@
+TITLE: GuixSD system tests
+
+From its inception, Guix has had a thorough test suiteâsomething thatâs not
only reassuring, but also the thing that allows for fearless evolution of the
code. That we didnât have this safety net when hacking on the whole operating
system, GuixSD, made it uncomfortable and more risky. We are now addressing
the problem with the introduction of _system tests_, closing one of the major
roadblocks towards 1.0.
+
+Before going into details, let me recap the sorts of testing that already
occurred in Guix land.
+
+== Unit tests ==
+
+Guixâs [http://git.savannah.gnu.org/cgit/guix.git/tree/tests test suite]
currently contains almost 600 _unit tests_. Each one of these stresses one
particular function or subset of the functionality of Guix. This covers core
package management functionality such as package builds, utility modules such
as monads or the public key infrastructure (PKI) used for authenticating
binaries, maintenance tools such as
[https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-lint.html li
[...]
+
+Since Guix provides Scheme modules for use
[https://www.gnu.org/software/guix/manual/html_node/G_002dExpressions.html both
in the package management front-end and on the âbuild sideâ], the latter is
also tested. This includes part of the
[https://www.gnu.org/software/guix/manual/html_node/Build-Systems.html build
systems], and helpers like our
[http://git.savannah.gnu.org/cgit/guix.git/tree/guix/build/gremlin.scm#n264 ELF
validation module].
+
+== Package tests ==
+
+Then come the software packages that Guix ships. All of the packages in the
distro are under [https://hydra.gnu.org/jobset/gnu/master continuous
integration] on the 4 supported architectures (32-bit and 64-bit Intel
compatible, as well as MIPS64 and ARMv7.) Our build farm serves the resulting
binaries, which users can choose to download as
[https://www.gnu.org/software/guix/manual/html_node/Substitutes.html
substitutes for local builds]. Our build server, which currently runs an insta
[...]
+
+Additionally, our policy is to always run each packageâs test suite (typically
âmake checkâ) as part of its build process, unless there is a serious technical
obstacle to doing that. That way, we can, and do catch integration issues,
incompatibilities, and plain bugs before they hit users.
+
+== System tests ==
+
+So far, so good. Now, what about GuixSD itself? GuixSD did not have an
automated test suite until now. What it did have, though, is the ability to
instantiate an operating system in a virtual machine (VM) or in a container.
You would write your
[https://www.gnu.org/software/guix/manual/html_node/Using-the-Configuration-System.html
operating system declaration] in a file, then run, say:
+
++verbatim+
+guix system vm my-config.scm
+-verbatim-
+
+This
[https://www.gnu.org/software/guix/manual/html_node/Invoking-guix-system.html#Invoking-guix-system
gives you a script to launch a VM] running an instance of the OS declared in
âmy-config.scmâ. Already pretty convenient! And indeed, even more so back in
the days when we were eating a fair amount of dog food. In fact, thatâs how we
ate our [https://savannah.gnu.org/forum/forum.php?forum_id=7506 first]
[https://savannah.gnu.org/forum/forum.php?forum_id=7737 dog food dishes], and
the [...]
+
+So what could we test exactly? Roughly, we want to test that the instantiated
system behaves according to the source âoperating-systemâ declaration: that
user accounts are all there, that system services are running as expected, that
all of the configuration is taken into account.
+
+To do that, we need to run the system under test in a VM, but we also need to
instrument it. We use [http://qemu.org QEMU] to run our VMs, and QEMU along
with the Linux virtio-serial module nicely supports communication between the
guest operating system and the host, a strategy also used by
[https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/test-driver/Machine.pm
NixOSâ test driver]. Concretely, we define a
[http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/build/marionette.scm [...]
+
+Now we can write build processes
([https://www.gnu.org/software/guix/manual/html_node/Derivations.html aka.
âderivationsâ]) that will:
+
+0 instantiate an instrumented variant of the operating system configuration we
want to test in a VM image;
+0 spawn the VM, run a series of tests on the guest OS, and return the test
results.
+
+Thus, a system test to make sure the
[https://www.gnu.org/software/guile/manual/html_node/System-Identification.html#index-scm_005funame
âunameâ] system call returns something that matches the OS declaration looks
like this:
+
++verbatim+
+(define (run-test)
+ (define os
+ ;; The declaration of the OS we want to instantiate and test.
+ ;; Calling 'marionette-operating-system' instruments it.
+ (marionette-operating-system
+ (operating-system
+ (host-name "komputilo")
+ (timezone "Europe/Berlin")
+ (locale "en_US.UTF-8")
+
+ (bootloader (grub-configuration (device "/dev/sdX")))
+ (file-systems %base-file-systems))))
+
+ ;; Compute the script to run OS in a VM.
+ (mlet %store-monad ((run (system-qemu-image/shared-store-script
+ os #:graphic? #f)))
+ (define test
+ ;; The actual test. Here â#~â is like âquoteâ, allowing us
+ ;; to describe code to run in the build environment; itâs a
+ ;; âg-expression.â
+ #~(begin
+ (use-modules (gnu build marionette)
+ (srfi srfi-64)
+ (ice-9 match))
+
+ (define marionette
+ ;; Spawn the VM that runs the declared OS.
+ (make-marionette (list #$run)))
+
+ (mkdir #$output)
+ (chdir #$output)
+
+ (test-begin "basic")
+
+ (test-assert "uname"
+ ;; Call the âunameâ Scheme function in the guest.
+ ;; In the host, make sure its result (a vector) matches
+ ;; our OS declaration above.
+ (match (marionette-eval '(uname) marionette)
+ (#("Linux" host-name version _ architecture)
+ (and (string=? host-name
+ #$(operating-system-host-name os))
+ (string-prefix? #$(package-version
+ (operating-system-kernel os))
+ version)
+ (string-prefix? architecture %host-type)))))
+
+ (test-end)
+ (exit (= (test-runner-fail-count (test-runner-current)) 0))))
+
+ ;; Turn the test into a buildable âderivationâ.
+ (gexp->derivation "simple-test" test
+ #:modules '((gnu build marionette)))))
+-verbatim-
+
+There are interesting things going on here. First, while this is all Scheme
code, there are in fact three tiers or strata of code at play here: the code
that produces the OS declaration and the derivation, the build code of that
derivationâthe testâembodied in a
[https://www.gnu.org/software/guix/manual/html_node/G_002dExpressions.html
g-expression], and code sent from the host to the guest VM via
âmarionette-evalâ.
+
+Using Scheme all the way means we can share code, use the right tools such as
the [http://srfi.schemers.org/srfi-64/srfi-64.html SRFI-64 test framework]
here, pass values from one tier to another, and so on. And of course, being a
full-blown language rather than shell scripts or similar means we have a rich
and semantically-clear interface at our fingertips: we can directly access the
data structures that matter rather than grepping the output of high-level
commands. As an example, we [...]
+
+== Status ==
+
+Guix now includes the test infrastructure described above; running âmake
check-systemâ runs all the currently defined tests. Currently we have
[http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/tests/base.scm tests for
basic functionality]. This includes making sure that all the system services
declared are available and running. We have tests for specific system services
such as the [https://www.gnu.org/software/mcron/ mcron] job scheduling
daemonâinspired by your parentsâ cron, but [...]
+
+Last but not least, we have
[http://git.savannah.gnu.org/cgit/guix.git/tree/gnu/tests/install.scm tests] of
the full
[https://www.gnu.org/software/guix/manual/html_node/System-Installation.html
GuixSD installation procedure], which turned out to be
[https://lists.gnu.org/archive/html/guix-devel/2016-06/msg00815.html more
involved] than the other tests. This works by running the GuixSD installation
image in a VM, using another VM image as the target installation media, and
finally bootin [...]
+
+All the tests are automatically run on our build farm (see
[https://hydra.gnu.org/job/gnu/master/test.basic.x86_64-linux here],
[https://hydra.gnu.org/job/gnu/master/test.installed-os.x86_64-linux here], or
[https://hydra.gnu.org/job/gnu/master/test.mcron.i686-linux there]), which
provides quick feedback. One step closer to 1.0!
+
+== About GNU Guix ==
+
+[http://www.gnu.org/software/guix GNU Guix] is a transactional package manager
for the GNU system. The Guix System Distribution or GuixSD is an advanced
distribution of the GNU system that relies on GNU Guix and
[http://www.gnu.org/distros/free-system-distribution-guidelines.html respects
the user's freedom].
+
+In addition to standard package management features, Guix supports
transactional upgrades and roll-backs, unprivileged package management,
per-user profiles, and garbage collection. Guix uses low-level mechanisms from
the Nix package manager, except that packages are defined as native
[http://www.gnu.org/software/guile Guile] modules, using extensions to the
[http://schemers.org Scheme] language. GuixSD offers a declarative approach to
operating system configuration management, and is [...]
+
+GuixSD can be used on an i686 or x86_64 machine. It is also possible to use
Guix on top of an already installed GNU/Linux system, including on mips64el and
armv7.