New kernel to play with

[Bas Wijnen]

Hello everyone,

You can consider this the end of my long break, although I'm not sure
how active I can be in the near future. :-)  But I have interesting
things to tell, so I shall not delay:

While waiting (very passively) for Coyotos to get ready, I decided that
I wanted to try writing a simple kernel myself.  It is based mostly on
the discussions I had here, most notably with Jonathan and Marcus.  It
was easier than I expected, which means it is running.  I also tried
porting glibc, but gave up on that.  Instead I wrote some minimal libc
myself (which is very non-optimized, but works mostly).  The current
state is such that it can run nethack (in text-mode).

For the impatient, here's how to see that (commands are for a Debian
system, adjust them if you have something else):

# apt-get install build-essential qemu automake1.10 autoconf git-core
# apt-get install rubber texlive-latex-base
$ git clone git://82.93.13.222/kernel
$ cd kernel
$ (mkdir nethack && cd nethack && apt-get source nethack)
$ ./autogen.sh
$ make
$ make qemu

Then when the system says "Waiting for events", type "run nethack" and
have fun. :-)

If it doesn't work, or you are more interested in the kernel than in
nethack, read on. :-)

First I'll tell a bit about the kernel.  There's a bit of outdated
documentation in doc/.  The most outdated facts are that capability
rights no longer exist, one numerical argument of invoke is changed into
a capability argument.  For the rest, it's incomplete.  I'll write an
explanation here as well.

At the moment, the kernel is ia-32 only.  I'd like it to work on other
cpus later, but that is not a priority.  I've tried a bit, but not too
hard, to keep the code portable.

The kernel has one system call, invoke, which is conceptually called on
a capability.  This call has 5 arguments: "request", "reply",
"argument", "a" and "b".  "request" and "argument" are numerical, the
others are capabilities.  When a process tries to send a capability it
doesn't own, NULL is sent instead.  The numerical arguments have no
restrictions.  The arguments have no special function, except when
calling kernel implemented objects.

I'll describe the kernel implemented objects one at a time.  First a
short list:
- space: this is the only item which can store things.  A space is
  always needed to do anything.  Any kernel object (including a space
  itself) needs a space to contain it.  The only exception to this is
  "first_space", which holds all memory not in use by the kernel.  The
  kernel never allocates any memory after it has started.

  I think it's very similar to the Coyotos space bank, although I don't
  know that well.  A major difference is that a space is in the kernel.

  If a space holds one or more processes, it also contains their mapping
  tables.

- process: (may be renamed to thread.)  A thread of execution.  This
  mostly contains the cpu state.  It might also contain scheduling
  priorities when there is a proper scheduler.  For now, everything
  which wants to run is scheduled in a very DoSsable way.  Also, there
  is no preemption (it is implemented but switched off), because that
  hinders debugging.

- userpage: a page of memory which can be used by userpsace processes.
  This is the only memory that can be mapped into view of userspace
  processes, all else is kernel only.

  Userpages can be shared and copied using copy-on-write.  The kernel
  handles the copying when a write occurs (conceptually this time is
  paid for by the writer, although no cpu time accounting takes place at
  the moment).

- receiver: a buffer which can be used to send messages to.  In
  particular, there are two ways to create capabilities: from a space
  about contained items (also on create), and creating them for a
  receiver.  When a capability for a receiver is invoked, a message is
  stored in the receiver's buffer.  A process can register itself as the
  receiver's owner and wait (or check) for new messages.  Usually only
  one receiver and process exist per space.  By using the protected
  value of the capabilities (set at creation), it is possible to know
  what type of request it is.

In order to see if this setup was usable, I implemented some userspace
parts.  There are drivers:

- keyboard.  Provides two slots where a capability can be registered.
  One for normal events (make and break codes, all repeat stuff is
  ignored), and one for "system request" (which because of IBM's
  defective design is the pause/break key, not
  print-screen/system-request).  The idea is to put the program which is
  "in focus" directly into the driver, without losing the sytem request
  supervisor ability.

- screen.  A simple vga text screen driver.  Allows setting the
  attribute, moving the cursor, and printing text (with the selected
  attribute).

- serial and parallel port.  Not really used at the moment, and not very
  complete (serial doesn't use fifo, parallel doesn't do interrupts).

And there are driver helpers:

- text: Map keyboard make events to ascii text.  This turns the keyboard
  driver into a "terminal" (together with screen).

- armour: A usb-style protected package protocol over the serial port.

And "normal" programs:

- armourfs: Use armour to serve a filesystem over a serial connection.

- elfload: You can send a space with a loaded file in here, and it will
  set up a process from it (static binaries only).

- ramfs: A ramdisk file system, including initial content (nethack, for
  example).

- test: The master testing program.  It sets up the drivers, and
  provides a simple shell with two commands: cat file to see contents of
  a file and run file args to run a file (with arguments).  When loaded,
  it provides this shell functionality to the keyboard/screen with ramfs
  and to the armour (one half of it) with armourfs (which uses the other
  half).  In the current state, the serial port and armour drivers are
  not loaded, though.

Finally, I wrote some libc stuff.  It's all in user/lib/ and does
exactly what you'd expect of it (except some things which don't do
anything at all).

***********************************************************************

So what can you do with it?

Well, I hope you can have some fun with it. :-)  It isn't really
intended to be the new Hurd kernel.  If you want to play with it, you
probably want to know what it doesn't do:

- It doesn't currently do persistence.  I think it may be possible to
  add this, but I haven't thought deeply about it.  There's currently no
  support for a proper backing store anyway (except armourfs, which is
  terribly slow).

- It doesn't have a proper scheduler.  This is easily solved.

- It doesn't currently have a proper way to handle page faults or other
  exceptions.  Some stuff is in place to stop the faulting process and
  send a request to somewhere, but it's not completely functional.  So
  far, I just let the kernel panic when a page fault happens (which is
  not a copy-on-write "request").

- It's missing lots of device drivers.  It's even missing interfaces for
  those drivers.

- The current libc implementation is very non-optimal and will need to
  be mostly replaced.  However, implementing fork and exec may be enough
  to run things like bash and all the non-graphical gnu tools.  (In
  other words, it may be non-optimal, but it is quite complete.)

- There are three ways to send data: in the invoke arguments (2 bytes),
  as a capability to a userpage (4kB) and as a capability to a space
  (unlimited).  It has been pointed out to me that the gap between 2
  bytes and 4kB is pretty big, and that updating the mapping for a page
  is very expensive if the interesting data is for example a printf
  output string.

If you want to play around, I suggest the following:
- first run nethack in the above steps, if you didn't yet.  Also note
  that you can press pause/break to abort at any time.
- you may want to run echo with some arguments as well.  This is a very
  simple program.
- read user/echo.c and user/screen.h.
- read kernel/kernel.h.  This is the interface between the kernel and
  userspace.
- read kernel/priv.h.  This is the interface between the kernel and
  device drivers (which run at protection level 0).
- read the keyboard and/or serial/parallel port driver.
- hack user/echo.c to make it do other things (as you can see, I did this
  myself already.  Originally, it was a normal echo).
- read user/test.c.

After that, you should have significant understanding of the kernel.
Whenever you feel like it, read things in kernel/, of course.  For your
information, the loader is the part which boots up the kernel and sets
everything up.  After that, it jumps to kernel/entry.c.  The interface
between the kernel and the loader (including most global variables) is
in loader/loader.h.  kernel/kernel-i.h and loader/loader-i.h are the
internal definitions of the kernel and loader respectively.

For the moment, any bug reports and comments can be sent to this list
(nothing else happens here anyway ;-) ).  If there is any continued
interest in the project, I'll set up some stuff somewhere else.

That should do for now, I wish you all a lot of fun with it.

Thanks,
Bas

-- 
I encourage people to send encrypted e-mail (see http://www.gnupg.org).
If you have problems reading my e-mail, use a better reader.
Please send the central message of e-mails as plain text
   in the message body, not as HTML and definitely not as MS Word.
Please do not use the MS Word format for attachments either.
For more information, see http://pcbcn10.phys.rug.nl/e-mail.html

signature.asc
Description: Digital signature