Re: Improving cgroups for fun and Kubernetes

[Ludovic Courtès]

Hi Tao,

Tao Hansen <worldofgeese@riseup.net> skribis:

> This is my second posting to the mailing list but the first using Gnus
> and smtmpmail. If I've formatted anything poorly, don't hesitate to let
> me know.

Looks perfect to me.  :-)

> I've been spending a silly amount of time trying to get a local flavor
> of Kubernetes running on Guix System. I wanted to share my experience
> and also solicit feedback from Guix's developers on how to improve the
> cgroups implementation such that those who follow me will have an easier
> time of it.

I’ve never used Kubernetes, but I’m confident you’re not the only
interested in using it on Guix System!

[...]

> The second problem is kind and minikube are both expecting Delegate=yes
> to be set, which is a systemd function that allows these tools to set
> cgroups limits. The limits it's expecting to control are cpu, cpuset,
> memory and pids. We can force these privileges like so, echo "+cpu
> +cpuset +memory +pids" >> /sys/fs/cgroup/cgroup.subtree_control

How about having a Shepherd service that does writes to that
‘cgroup.subtree_control’ file as you write above?

> To fix the first problem we can run
>
> g=users && sudo chgrp -R ${g} /sys/fs/cgroup/
> u=$USER && sudo chown -R ${u}: /sys/fs/cgroup

What does Debian do?  Perhaps there’s a “cgroup” group (in /etc/groups)
that users who want to user podman need to belong to, similar to the
‘kvm’ group for those who want to access /dev/kvm?

Or maybe we should create a sub-tree specifically for podman usage?

At any rate, we could again have a Shepherd service that sets ownership
on the relevant file tree.

> Once we've addressed the first and second problem, the rest is
> relatively easy: we need to make iptables (and iptables' modules so just
> the package isn't enough: we need Guix's service) available. We need to
> set a range of user IDs and group IDs for Podman to make use of
> rootlessly, and finally we need to set a container policy otherwise Podman
> can't pull any image from anywhere. All of those can be done from inside
> our Guix System configuration file.

Right, we should populate /etc/subuid by default (I tried to use
subordinate UIDs in the past, by invoking ‘newuidmap’, but never managed
to get it to work.)

> Here's what that Guix System configuration looks like:
>
> ;; Rootless Podman requires the next 4 services
> ;; we're using the iptables service purely to make its resources
> ;; available to minikube and kind
>
> (service iptables-service-type
>          (iptables-configuration
>           (ipv4-rules (plain-file "iptables.rules" "*filter
> :INPUT ACCEPT
> :FORWARD ACCEPT
> :OUTPUT ACCEPT
> COMMIT
> "))
>               (ipv6-rules (plain-file "ip6tables.rules" "*filter
> :INPUT ACCEPT
> :FORWARD ACCEPT
> :OUTPUT ACCEPT
> COMMIT
> "))))
>       (simple-service 'etc-subuid etc-service-type
>                       (list `("subuid" ,(plain-file "subuid"
>          (string-append "root:0:65536\n" username ":100000:65536\n")))))
>       (simple-service 'etc-subgid etc-service-type
>                       (list `("subgid" ,(plain-file "subgid"
>          (string-append "root:0:65536\n" username ":100000:65536\n")))))
>     (service pam-limits-service-type
>              (list
>               (pam-limits-entry "*" 'both 'nofile 100000)))
>     (simple-service 'etc-container-policy etc-service-type
>                       (list `("containers/policy.json", (plain-file
>          "policy.json" "{\"default\": [{\"type\":
>          \"insecureAcceptAnything\"}]}"))))
>     %my-services

Looks great!  We should probably consider /etc/{subuid,subgid} support
separately, but otherwise it looks like you already have the start of a
‘rootless-podman-service-type’ (or similar).

Thanks,
Ludo’.