[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[bug#42261] [PATCH] website: Add draft of a Ganeti cluster post.
|
From: |
Marius Bakke |
|
Subject: |
[bug#42261] [PATCH] website: Add draft of a Ganeti cluster post. |
|
Date: |
Wed, 8 Jul 2020 12:11:18 +0200 |
From: Marius Bakke <mbakke@fastmail.com>
* website/drafts/ganeti-cluster-on-guix.md: New file.
---
website/drafts/ganeti-cluster-on-guix.md | 414 +++++++++++++++++++++++
1 file changed, 414 insertions(+)
create mode 100644 website/drafts/ganeti-cluster-on-guix.md
diff --git a/website/drafts/ganeti-cluster-on-guix.md
b/website/drafts/ganeti-cluster-on-guix.md
new file mode 100644
index 0000000..253681b
--- /dev/null
+++ b/website/drafts/ganeti-cluster-on-guix.md
@@ -0,0 +1,414 @@
+title: Running a Ganeti cluster on Guix
+date: 2020-07-10 12:00
+author: Marius Bakke
+tags: Virtualization, Ganeti
+---
+The latest addition to Guix's ever-growing list of services is a little-known
+virtualization toolkit called [Ganeti](http://www.ganeti.org/). Ganeti is
+designed to keep virtual machines running on a cluster of servers even in the
+event of hardware failures, and to make maintenance and recovery tasks easy.
+
+It is comparable to tools such as
+[Proxmox](https://www.proxmox.com/en/proxmox-ve) or
+[oVirt](https://www.ovirt.org/), but has some distinctive features. One is
+that there is no GUI: [third](https://github.com/osuosl/ganeti_webmgr)
+[party](https://github.com/grnet/ganetimgr)
+[ones](https://github.com/sipgate/ganeti-control-center) exist, but are not
+currently packaged in Guix, so you are left with a rich command-line client
+and a fully featured
+[remote API](http://docs.ganeti.org/ganeti/master/html/rapi.html).
+
+Another interesting feature is that installing Ganeti on its own leaves you
+no way to actually deploy any virtual machines. That probably sounds crazy,
+but stems from the fact that Ganeti is designed to be API-driven and automated,
+thus it comes with a
+[OS
API](http://docs.ganeti.org/ganeti/master/html/man-ganeti-os-interface.html)
+and users need to install one or more *OS providers* in addition to Ganeti.
+OS providers offer a declarative way to deploy virtual machine variants and
+should feel natural to Guix users. At the time of writing, the providers
+available in Guix are
[debootstrap](https://github.com/ganeti/instance-debootstrap)
+for provisioning Debian- and Ubuntu-based VMs, and of course a
+[Guix](https://github.com/mbakke/ganeti-instance-guix) provider.
+
+Finally Ganeti comes with a sophisticated scheduler that efficiently packs
+virtual machines across a cluster while maintaining N+1 redundancy in case
+of a failover scenario. It can also make informed scheduling decisions
+based on various cluster tags, such as ensuring primary and secondary nodes
+are on different power distribution lines.
+
+(Note: if you are looking for a way to run just a few virtual machines on
+your local computer, you are probably better off using
+[libvirt](https://guix.gnu.org/manual/en/guix.html#index-libvirt) or even
+a
[Childhurd](https://guix.gnu.org/manual/devel/en/guix.html#index-hurd_002dvm_002dservice_002dtype),
as Ganeti is fairly heavyweight and requires a complicated networking
+setup.)
+
+
+# Preparing the configuration
+
+With introductions out of the way, let's see how we can deploy a Ganeti
+cluster using Guix. For this tutorial we will create a two-node cluster
+and connect instances to the local network using an
+[Open vSwitch](https://www.openvswitch.org/) bridge with no VLANs. We assume
+that each node has a single network interface named `eth0` connected to the
+same network, and that a dedicated partition `/dev/sdz3` is available for
+virtual machine storage. It is possible to store VMs on a number of other
+storage backends, but a dedicated drive (or rather LVM volume group) is
+necessary to use the [DRBD](https://www.linbit.com/drbd/) integration to
+replicate VM disks.
+
+We'll start off by defining a few helper services to create the Open vSwitch
+bridge and ensure the physical network interface is in the "up" state. Since
+Open vSwich stores the configuration in a database, you might as well run the
+equivalent `ovs-vsctl` commands on the host once and be done with it, but we
+do it through the configuration system to ensure we don't forget it in the
+future when adding or reinstalling nodes.
+
+```
+(define (start-interface if)
+ #~(let ((ip (string-append #$iproute "/sbin/ip")))
+ (invoke/quiet ip "link" "set" #$if "up")))
+
+(define (stop-interface if)
+ #~(let ((ip (string-append #$iproute "/sbin/ip")))
+ (invoke/quiet ip "link" "set" #$if "down")))
+
+;; This service is necessary to ensure eth0 is in the "up" state on boot
+;; since it is otherwise unmanaged from Guix PoV.
+(define (ifup-service if)
+ (let ((name (string-append "ifup-" if)))
+ (simple-service name shepherd-root-service-type
+ (list (shepherd-service
+ (provision (list (string->symbol name)))
+ (start #~(lambda ()
+ #$(start-interface if)))
+ (stop #~(lambda ()
+ #$(stop-interface if)))
+ (respawn? #f))))))
+
+(define* (create-openvswitch-bridge bridge uplink
+ #:key (vlan-mode #f))
+ #~(let ((ovs-vsctl (lambda (cmd)
+ (apply invoke/quiet
+ #$(file-append openvswitch "/bin/ovs-vsctl")
+ (string-tokenize cmd)))))
+ (and (ovs-vsctl (string-append "--may-exist add-br " #$bridge))
+ (ovs-vsctl (string-append "--may-exist add-port " #$bridge " "
+ #$uplink
+ (if #$vlan_mode
+ (format #f " vlan_mode=~a "
#$vlan-mode)
+ ""))))))
+
+(define* (create-openvswitch-internal-port bridge port
+ #:key (vlan-mode #f))
+ #~(invoke/quiet #$(file-append openvswitch "/bin/ovs-vsctl")
+ "--may-exist" "add-port" #$bridge #$port
+ (if #$vlan_mode
+ (string-append "vlan_mode=" #$vlan-mode)
+ "")
+ "--" "set" "Interface" #$port "type=internal"))
+
+(define %openvswitch-configuration-service
+ (simple-service 'openvswitch-configuration shepherd-root-service-type
+ (list (shepherd-service
+ (provision '(openvswitch-configuration))
+ (requirement '(vswitchd))
+ (start #~(lambda ()
+ #$(create-openvswitch-bridge
+ "br0" "eth0"
+ #:vlan_mode "native-untagged")
+ #$(create-openvswitch-internal-port
+ "br0" "gnt0"
+ #:vlan_mode "native-untagged")))
+ (respawn? #f)))))
+```
+
+This defines a `openvswitch-configuration` service object that creates a
+logical switch `br0`, connects `eth0` as the "uplink", and creates a logical
+port `gnt0` that we will use later as the main network interface for this
+system. We also create an `ifup` service that can bring network interfaces
+up and down. By themselves these variables do nothing, we also have to add
+them to our `operating-system` configuration below.
+
+A configuration like this might be suitable for a small home network. In most
+"real world" deployments you would use tagged VLANs, and maybe a traditional
+Linux bridge instead of Open vSwitch. You can also forego bridging altogether
+with a `routed` networking setup, or do any combination of the three.
+
+With this in place, we can start creating the `operating-system` configuration
+that we will use for the Ganeti servers:
+
+```
+(operating-system
+ (host-name "node1")
+ [...]
+ ;; Ganeti requires that each node and the cluster address resolves to an
+ ;; IP address. The easiest way to achieve this is by adding everything
+ ;; to the hosts file.
+ (hosts-file (plain-file "hosts" (format #f "\
+127.0.0.1 localhost
+::1 localhost
+
+192.168.1.101 node1
+192.168.1.102 node2
+192.168.1.254 ganeti.lan
+")))
+ (kernel-arguments
+ (append %default-kernel-arguments
+ '(;; Disable DRBDs usermode helper, as Ganeti
+ ;; is the only thing that should manage DRBD.
+ "drbd.usermode_helper=/run/current-system/profile/bin/true")))
+
+ (packages (append (map specification->package
+ '("qemu" "drbd-utils" "lvm2"
+ "ganeti-instance-guix"
+ "ganeti-instance-debootstrap"))
+ %base-packages))
+
+ (services (cons* (service ganeti-service-type
+ (ganeti-configuration
+ (file-storage-paths '("/srv/ganeti/file-storage"))
+ (os
+ (list (ganeti-os
+ (name "debootstrap")
+ (variants
+ (list (debootstrap-variant
+ "buster"
+ (debootstrap-configuration
+ (hooks
+ (local-file
+ "debootstrap-hooks"
+ #:recursive? #t))))
+ (debootstrap-variant
+ "testing+contrib"
+ (debootstrap-configuration
+ (suite "testing")
+ (components '("main"
"contrib")))))))))))
+
+ ;; Create a static IP on the "gnt0" Open vSwitch interface.
+ (service openvswitch-service-type)
+ %openvswitch-configuration-service
+ (ifup-service "eth0")
+ (static-networking-service "gnt0" "192.168.1.101"
+ #:netmask "255.255.255.0"
+ #:gateway "192.168.1.1"
+ #:requirement
'(openvswitch-configuration)
+ #:name-servers '("192.168.1.1"))
+
+ ;; Ganeti needs SSH to communicate between nodes.
+ (service openssh-service-type
+ (openssh-configuration
+ (permit-root-login 'without-password)))
+ %base-services)))
+```
+
+Debootstrap variants rely on a set of scripts (known as "hooks") in the
+installation process to do things like configure networking, install
bootloader,
+create users, etc. In the example above, the "buster" variant will use a local
+directory next to the configuration file named "debootstrap-hooks" (it is
copied
+into the final system closure), whereas the "testing+contrib" variant has no
hooks
+defined and will use `/etc/ganeti/instance-debootstrap/hooks` if it exists.
+
+Ganeti veterans may be surprised that each OS variant has its own hooks. All
+Ganeti clusters I know of use a single set of hooks for all variants, sometimes
+with additional logic inside the script based on the variant. Guix offers a
+powerful abstraction that makes it trivial to create per-variant hooks,
obsoleting
+the need for a big `/etc/ganeti/instance-debootstrap/hooks` directory. Of
course
+you can still create it using `extra-special-file` and leave the `hooks`
property
+of the variants as `#f`.
+
+Not all Ganeti options are exposed in the configuration system yet. If you
+find it limiting, you can add custom files using `extra-special-file`, or
+ideally extend the `<ganeti-configuration>` data type to suite your needs.
+Of course you can use `gnt-cluster copyfile` and `gnt-cluster command`
+to distribute files or run executables, but beware that undeclared changes
+in `/etc` may be lost on the next reboot or reconfigure.
+
+
+# Initializing a cluster
+
+At this stage, you should run `guix system reconfigure` with the new
+configuration on all nodes that will participate in the cluster. If you
+do this over SSH or with
+[guix
deploy](https://guix.gnu.org/blog/2019/managing-servers-with-gnu-guix-a-tutorial/),
+beware that `eth0` will lose network connectivity once it is "plugged in to"
+the virtual switch, and you need to add any IP configuration to `gnt0`.
+
+The Guix configuration system does not currently support declaring LVM
+volume groups, so we will create these manually on each node. We could
+write our own declarative configuration like the `ifup-service`, but for
+brevity and safety reasons we'll do it "by hand":
+
+```
+pvcreate /dev/sdz3
+vgcreate ganetivg /dev/sdz3
+```
+
+On the node that will act as the "master node", run the init command:
+
+```
+gnt-cluster init \
+ --master-netdev=gnt0 \
+ --vg-name=ganetivg \
+ --enabled-disk-templates=file,plain,drbd \
+ --drbd-usermode-helper=/run/current-system/profile/bin/true \
+ --enabled-hypervisors=kvm \
+ --no-etc-hosts \
+ --no-ssh-init \
+ ganeti.lan
+```
+
+If you are okay with Ganeti taking control over SSH `authorized_keys` and
+`known_hosts`, remove the `--no-ssh-init` option. Guix users might prefer
+to manage the relevant files using `openssh-configuration`. All nodes in
+the cluster must be able to reach each other over SSH as the root user.
+
+Similarly, Ganeti can update the `/etc/hosts` file when nodes are added or
+removed, but it makes little sense on Guix as it is recreated every reboot.
+
+If all goes well, the command returns no output and you should have the
+`ganeti.lan` IP address visible on `gnt0`. You can run `gnt-cluster verify`
+to check that the cluster is in good shape. Most likely it complains about
+something:
+
+```
+# TODO
+```
+
+Use `gnt-cluster modify` to change the running state of the cluster:
+
+```
+gnt-cluster modify -H kvm:kernel_path=
+```
+
+The command above removes the warning about the default KVM kernel being
+missing, making `gnt-cluster verify` happy. For this tutorial we only use
+fully virtualized instances, but users might want to set `kernel_path` to a
+suitable VM kernel.
+
+Now let's add our other machine to the cluster:
+
+```
+gnt-node add node2
+```
+
+Ganeti will log into the node, copy the cluster configuration and start the
+relevant Shepherd services. No output means the command succeeded. Run
+`gnt-cluster verify` again to check that everything is in order:
+
+```
+gnt-cluster verify
+```
+
+If you get warnings about SSH authorizations here, you should fix those
+before proceeding. If you used `--no-ssh-init` earlier you may need to
+update `/var/lib/ganeti/known_hosts` with the new node information, either
+with `gnt-cluster copyfile` or by adding it to the OS configuration.
+
+The above configuration will make three operating systems available:
+
+```
+# gnt-os list
+Name
+guix
+debootstrap+buster
+debootstrap+testing+contrib
+```
+
+Let's try them out. But first we'll make Ganeti aware of our network
+so it can choose a static IP for the virtual machines.
+
+```
+# gnt-network add --network=192.168.1.0/24 --gateway=192.168.1.1 lan
+# gnt-network connect -N mode=openvswitch,link=br0 lan
+```
+
+Now we can add an instance:
+
+```
+gnt-instance add --no-name-check --no-ip-check -o debootstrap+buster \
+ -t drbd --disk 0:size=5G -B memory=256m,vcpus=2 \
+ --net 0:network=lan,ip=pool bustervm1
+```
+
+Ganeti will automatically select the optimal primary and secondary node
+for this VM based on available cluster resources. You can manually
+specify primary and secondary nodes with the `-n` and `-s` options.
+
+By default Ganeti assumes that the new instance is already configured in DNS,
+so we need `--no-name-check` and `--no-ip-check` to bypass some sanity tests.
+
+Try adding another instance, now using the Guix OS provider:
+
+```
+gnt-instance add --no-name-check --no-ip-check -o guix \
+ -t plain --disk 0:size=5G -B memory=1G,vcpus=4 \
+ --net 0:network=lan,ip=pool guix1
+```
+
+The Guix OS has a built-in configuration that starts an SSH server and
authorizes
+the hosts SSH key, and configures static networking based on information from
+Ganeti. It is possible to specify a custom configuration file, and even a
+specific Guix commit:
+
+```
+gnt-instance add --no-name-check --no-ip-check -o guix \
+ -t file --file-storage-dir=/srv/ganeti/file-storage \
+ --disk 0:size=20G -B memory=4G,vpus=3 \
+ --net 0:network=lan,ip=pool \
+ -O "config=$(base64 /the/config/file.scm),commit=<commit>" \
+ custom-guix
+```
+
+That's it for this tutorial! If you are new to Ganeti, you should
+familiarize yourself with the `gnt-` family commands. Fun stuff to
+do include `gnt-instance migrate` to move VMs between hosts,
+`gnt-node evacuate` to migrate _all_ VMs off a node, and
+`gnt-cluster master-failover` to move the master role to a different node.
+
+
+# Final remarks
+
+Like most services in Guix, Ganeti comes with a
+[system test](https://guix.gnu.org/blog/2016/guixsd-system-tests/)
+that [runs in a VM](FIXME) and ensures that things like initializing a cluster
+work. The continuous integration system
+[runs this automatically](https://ci.guix.gnu.org/search?query=ganeti), and
+users can run it locally with `make check-system TESTS=ganeti`. Such
+tests give us confidence that both the package and configuration system work,
+and allows rapid testing of the configuration API. Currently it does little
+more than `gnt-cluster verify`, but it can be extended to provision a real
+cluster inside Ganeti and try things like live migration.
+
+The author had a lot of fun creating
+[native data types](FIXME manual link)
+in the Guix configuration system for the Ganeti OS specification. The API
+went through at least three major revisions during the writing of this blog
+post. There is still room for improvement, but I decided I had to stop
+tweaking it and instead focus on shipping the thing. Feedback welcome!
+
+Having OS support in the configuration system lets us benefit from Guix's
+provenance tracking and we can easily `guix system roll-back` any breaking
+changes. Ganeti is usually coupled with tools such as Puppet or SaltStack to
+keep things in sync between nodes, but that should not be necessary here.
+
+So far only the `KVM` hypervisor has been tested. If you use LXC or Xen with
+Ganeti, please reach out to `guix-devel@gnu.org` and share your experience.
+
+#### About GNU Guix
+
+[GNU Guix](https://guix.gnu.org) is a transactional package
+manager and an advanced distribution of the GNU system that [respects
+user
+freedom](https://www.gnu.org/distros/free-system-distribution-guidelines.html).
+Guix can be used on top of any system running the kernel Linux, or it
+can be used as a standalone operating system distribution for i686,
+x86_64, ARMv7, and AArch64 machines.
+
+In addition to standard package management features, Guix supports
+transactional upgrades and roll-backs, unprivileged package management,
+per-user profiles, and garbage collection. When used as a standalone
+GNU/Linux distribution, Guix offers a declarative, stateless approach to
+operating system configuration management. Guix is highly customizable
+and hackable through [Guile](https://www.gnu.org/software/guile)
+programming interfaces and extensions to the
+[Scheme](http://schemers.org) language.
--
2.27.0