make-rc: A parallel (as in make(1)) alternative to sysv-rc

[Alejandro Colomar (man-pages)]

Hi all,

Most of you I added you to this email because I found you on the 
maintainers list for the Debian sysv-rc package (now dead for a long time).
I also CCd Devuan, since I hope you'll be interested in this little 
project of mine.
I also CCd linux-man@, since there's not many people listening there, 
and not much traffic these days so I hope you won't be angry for this 
little bit of spam; and I hope some good guys reading that list may have 
some comments on this.  Sorry again for the bit of spam.
I also CCd help-make@.  I (ab)used your software in a way that it was 
never designed to; not sorry for that; I'll say it was the original Unix 
authors' fault for designing such (ab)usable tools :).  Maybe you're 
interested in this thread, maybe you don't care, but I'll CC you in case 
some of you may be interested in this.
And finally, Randy, as you asked, I'll CC you for news on this.
Anyone not interested in follow-ups, please email me, and I'll try to 
remove from CC.

So, last friday (yes, that's New Year's Eve), I was reading something, 
and got this idea... the main valid claim for systemd is that it blows 
away competition in terms of performance?  Full parallelization?  Knows 
about dependencies?  I don't know too much of systems; I know some basic 
stuff, but I'm mostly a C programmer, so I don't know init(1)... okay. 
But since I program, I sure know the good ol' make(1).  It's old, and 
it's good at fully parallelizing _everything_.  Dependencies? sure; 
fast? sure; parallel? sure; simple? sure; a bit bloated? GNU make maybe, 
especially for init(1), but far from systemd(1):

$ ls -lh $(realpath $(which systemd make bash sh 2>/dev/null))
-rwxr-xr-x 1 root root 1.2M Dec 15 00:43 /usr/bin/bash
-rwxr-xr-x 1 root root 123K Nov  3 11:51 /usr/bin/dash
-rwxr-xr-x 1 root root 235K Apr 10  2021 /usr/bin/make
-rwxr-xr-x 1 root root 1.8M Nov 19 21:11 /usr/lib/systemd/systemd

So, if the problem is that the rc scripts don't run parallel and don't 
know about exact dependencies from each-other, let's rewrite that part 
and only that part of the system with make(1); that was the idea.  Don't 
touch init, don't touch the scripts themselves... only the part that 
decides on which order to run them.  That's the idea.


I decided to start with a clean install of Devuan (since I'm a Debian 
user, and I'm used to it; it might have been easier to start with a 
smaller system, but I might have found other issues in the middle;  I 
know that the only difference in Devuan is sysvinit, which is what I 
want).  The install has XFCE and sysvinit as options.  I wanted the 
system to have GUI, so that if I can boot XFCE with it, I can boot 
anything (except maybe for the bad boy, GNOME, but I don't even like it).

I wrote a couple of scripts to port the existing rc system to my 
make-based rc system.  The script didn't touch the old one, in case I 
needed it to boot if I screwed something, of course.  One script creates 
the makefiles without declaring the dependencies between them, and then 
another makefile creates the dependencies to match the current boot 
order (allowing for the same level of parallelization that the current 
semi-parallel rc script uses).  From there, one could analyze the 
dependencies to remove some that are incorrect, and make it faster, but 
since I don't know the real dependencies, I didn't want to break stuff.

So, here go the scripts:

====================== script1_basicmk.sh ========================
#!/bin/bash

find /etc/rc[0123456S].d/[KS]* \
|sed s,/etc/rc..d/...,, \
|sort \
|uniq \
|while read x; do\
        mk="/etc/rc.mk.d/$x.mk";

        test -e $mk \
                && continue;

        >$mk cat <<EOF
.PHONY: K$x S$x

: K$x

: S$x

K$x:
        /etc/init.d/$x stop

S$x:
        /etc/init.d/$x start
EOF

        find /etc/rc[0123456S].d/K*$x 2>/dev/null \
        |sed s,/etc/rc,, \
        |sed s,.d/K.*,, \
        |sort \
        |tac \
        |while read n; do \
                sed -i "/: K$x$/s/^/$n /" $mk;
        done;

        find /etc/rc[0123456S].d/S*$x 2>/dev/null \
        |sed s,/etc/rc,, \
        |sed s,.d/S.*,, \
        |sort \
        |tac \
        |while read n; do \
                sed -i "/: S$x$/s/^/$n /" $mk;
        done;

        sed -i "s/ *:/:/" $mk;
        sed -i "/^:/d"    $mk;
done;
=========================================================

===================== script2_deps.sh ===================
#!/bin/bash

find /etc/rc.mk.d/*.mk \
|sed s,/etc/rc.mk.d/,, \
|sed s,.mk$,, \
|sort \
|while read x; do\
        mk="/etc/rc.mk.d/$x.mk";

        for n in 0 1 2 3 4 5 6 S; do \
                K=$(find /etc/rc$n.d/K??$x 2>/dev/null \
                    |grep -o /K.. \
                    |sed s,/K0*,,);
                test -z "$K" || test $K -eq 1 \
                        && continue;
                k=$(printf %02d $(($K - 1)));

                find /etc/rc$n.d/K$k* \
                |sed s,/etc/rc$n.d/K$k,,;
        done \
        |sort \
        |uniq \
        |while read dep; do \
                echo K$dep;
        done \
        |xargs echo \
        |while read Kdeps; do
                test -z "$Kdeps" \
                        && break;

                >>$mk cat <<-EOF

                K$x: $Kdeps
                EOF
        done;

        for n in 0 1 2 3 4 5 6 S; do \
                S=$(find /etc/rc$n.d/S??$x 2>/dev/null \
                    |grep -o /S.. \
                    |sed s,/S0*,,);
                test -z "$S" || test $S -eq 1 \
                        && continue;
                s=$(printf %02d $(($S - 1)));

                find /etc/rc$n.d/S$s* \
                |sed s,/etc/rc$n.d/S$s,,;
        done \
        |sort \
        |uniq \
        |while read dep; do \
                echo S$dep;
        done \
        |xargs echo \
        |while read Sdeps; do
                test -z "$Sdeps" \
                        && break;

                >>$mk cat <<-EOF

                S$x: $Sdeps
                EOF
        done;

        for n in 0 1 2 3 4 5 6 S; do \
                test -e /etc/rc$n.d/S01$x \
                        || continue;
                K=$(find /etc/rc$n.d/K* 2>/dev/null \
                    |grep -o /K.. \
                    |sed s,/K,, \
                    |sort \
                    |uniq \
                    |tac \
                    |head -n1)
                test -z "$K" \
                        && continue;
                k=$(printf %02d $K);

                find /etc/rc$n.d/K$k* \
                |sed s,/etc/rc$n.d/K$k,,;
        done \
        |sort \
        |uniq \
        |while read dep; do \
                echo K$dep;
        done \
        |xargs echo \
        |while read Kdeps; do
                test -z "$Kdeps" \
                        && break;

                >>$mk cat <<-EOF

                S$x: $Kdeps
                EOF
        done;
done;
===============================================


I know some stuff could go into functions to make them shorter, and 
probably some comments could also help, but I wrote them fast, just to 
have some proof of concept, and later I could makes them nicer.  Special 
thanks to Doug for creating pipes, I love them :)

The One Makefile To Rule Them All is the following:

================ /etc/rc.mk.d/Makefile ================
MAKEFLAGS += --no-builtin-rules
MAKEFLAGS += --no-builtin-variables

.PHONY: all 0 1 2 3 4 5 6 S

all: 5

0 1 2 3 4 5 6 S:

include /etc/rc.mk.d/*.mk
=======================================================

And then a sample file (created by the above scripts) is:

================ /etc/rc.mk.d/vboxadd.mk ===============
.PHONY: Kvboxadd Svboxadd

0 1 6: Kvboxadd

2 3 4 5: Svboxadd

Kvboxadd:
        /etc/init.d/vboxadd stop
        sleep 1

Svboxadd:
        /etc/init.d/vboxadd start
        sleep 1

Kvboxadd: Kalsa-utils Kbrightness Kelogind Knetwork-manager 
Kpulseaudio-enable-autospawn Ksaned Kslim Kspeech-dispatcher Kurandom 
Kvboxadd-service

Svboxadd: Sconsole-setup.sh
========================================================

I added the sleep 1 for debug; the rest is the output of the scripts.


As you can see I kept the K... and S... syntax of rc scripts.  However, 
instead of a symlink, now there's a makefile declaring dependencies and 
calls directly the script with no symlink.  Now all makefiles are in a 
single directory (I found it simpler this way), but we could think of a 
different structure if it's better.  These makefiles (or more 
technically, parts of a makefile), resemble the systemd service files, 
where each one declares what should have already run before start and 
before kill, and also contains the info about which runlevels want this 
(both S and K).

Because of the script, and because some scripts used .sh extension and 
some others didn't, there are some traces of that in the names of the 
targets; nothing really problematic, only ugly.  I could have removed 
it, but it was more work, and I noticed late.

I only had to manipulate a single line from only one of these makefiles 
after running the scripts, but that I'll keep if for later;  it was 
about dependencies, because there was a missing number in one of the 
runlevel dirs (no jump from S02 to S04 in one of the dirs, no S03)


And finally, to run all this madness, I had to tweak /lib/init/rc a 
little bit:

diff --git a/rc b/rc
index 5e7f3a4..e1ffa31 100755
--- a/rc
+++ b/rc
@@ -60,7 +60,7 @@ export VERBOSE
 # insserv package to be enabled. Boot concurrency also requires
 # startpar to be installed.
 #
-CONCURRENCY=makefile
+CONCURRENCY=makefile2
 test -s /etc/init.d/.depend.boot  || CONCURRENCY="none"
 test -s /etc/init.d/.depend.start || CONCURRENCY="none"
 test -s /etc/init.d/.depend.stop  || CONCURRENCY="none"
@@ -168,7 +168,10 @@ then
                done
        fi

-       if [ makefile = "$CONCURRENCY" ]
+       if [ makefile2 = "$CONCURRENCY" ]
+       then
+               make -C "/etc/rc.mk.d/" "$runlevel"
+       elif [ makefile = "$CONCURRENCY" ]
        then
                if [ S = "$runlevel" ]
                then



I'd like to know your thoughts about this.  If it seems like it could be 
a good replacement for sysv-rc (for Devuan; I guess for Debian it's too 
late) I'd like to know.  Or if you think it can be a good thing but 
needs some polish, I'd like to hear it too.  It's just a proof of 
concept after 2 days of having had fun with it :)

I'd like it to be able to compete in terms of performance with 
systemd(1), so that we could possibly take over it, but hey, that's only 
a hope.

And now I'll tell you a bit about the outcome I had:  I booted with this 
configuration, and I first got into single-user mode, which I don't yet 
understand why it happened (there's probably something I didn't get 
right, but no real blocking problem, I guess).  But then, after the 
typical type password or ^D to continue, I pressed ^D, and I got a shiny 
XFCE GUI completely wirking.  Not bad for this prototype, I'd say.  I'd 
like some help to polish this, if you may.

Happy new year!

Alex


--
Alejandro Colomar
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
http://www.alejandro-colomar.es/