TODO List v0.00

[Farid Hajji]

This is zero-th try at a TODO list for the Hurd/L4 port.
Contributions and comments highly desired. I won't happen
without your help.

L4-Area:
--------

Documentation and admin stuff:
..............................

* Set up a l4-hurd project page somewhere (preferably on
  sourceforge.net or gnu.org). Needed here:
    * Short intro, explaining the reasons for porting the
      Hurd from Mach to L4 (or to VK).
    * Links to the canonical L4 sites Dresden, Karlsruhe
      and UNSW.
    * Links to the archived mailing lists l4-hackers, l4ka,
      and of course l4-hurd.
    * Download area with instructions on getting at least:
        Fiasco, L4Linux, OSKit
        Hazelnut, ChacmOS
        Instruction manual from UNSW
    * Set up a CVS repository and add a link here.
  Information about this can be found on the l4-hurd archives,
  and the Dresden, Karlsruhe and UNSW sites. To clarify points,
  please mail to l4-hurd.

* Write a good introductory text on L4 programming. This text
  should at least contain the following topics:
    * General L4 infos, as found in Jochen's et. al. papers.
      Why L4, Tasks, Threads, IPC (kinds of), Sigma0, etc...
    * C API for accessing L4 primitives. Here, the User's
      Manual from UNSW would be a good starting point.
      It should be adapter to Hazelnut/x86 though, so
      that more people could use it directly.
    * Writing root-tasks (Lot's of examples: starting/stopping
      threads, forking off tasks, setting up pagers, lot's of
      raw IPC, etc...).
      Here, look at L4Linux' L4-arch code and at ChacmOS
      for ideas and sample code.

L4 user-land hacking
....................

* Start by writing some root-tasks for L4 (e.g. Hazelnut), that
  exercise the L4 C-API. Create and manage some threads, activate
  new tasks, do inter-task IPCs (short, long etc...). Get used
  to the L4 C-API(s).

* Write (or adapt) a generic default pager that would serve as a
  general VM handler. This pager should back memory requests
  from every OS personality that we'll want to run on top of L4,
  but also from naked tasks.

  Reusing code is possible here: Mach's default pager may be
  a good starting point, but there are better alternatives
  like UVM. As an excercise, you'll want to write a toy-
  pager first, but that is strictly not necessary.

  Since we don't have disks at this stage, simply write dirty
  pages into a memory area reserved by the pager (page to memory)
  so far. This limits the amount of available memory to 1/2,
  but this restriction will be lifted later ;)

  Write an API to access the pager directly (IDL specs?!). The
  way Mach specifies this seems a good starting point (but don't
  get too fancy here). Have vm_map(), vm_allocate(), vm_deallocate()
  and vm_remap() use this API, so that we can use malloc() on top
  of vm_*().

  Adapt OSKit memory routines accordingly [opt.].

* Write some simple device drivers to access the hardware directly.
  I'd suggest to use Mach abstractions/APIs here as far as
  client programs are concerned.

  This is not for the faint of heart. You'll need to understand
  how L4 provides access to
   * physical memory addresses (pager will need to map 1:1 from sigma0),
   * physical i/o addresses (on x86, i/o and memory space are distinct),
   * interrupt notifications (IPC receive on l4-thread).

  One possible architecture would be to write a meta-driver task,
  that accesses the hardware through L4 and then one task per
  driver would IPC from/to this meta-driver to get hardware resources.
  driver tasks could register for interrupt (notifications), and could
  map (parts of) the i/o address and memory address space, so that they
  can access their hardware directly.

  The main advantage of this two-layer architecture is to provide a
  "docking area" for common device drivers, as well as for managing
  resource sharing (such as DMA, IRQs, Memory regions etc...) among
  other disparate drivers.  [Explanation: drivers from one OS often
  disable interrupts in their non-reentrant sections. This scheme is
  not portable across drivers from different OSes. Furthermore, we
  can't do this on L4. So we need to have a "DMGR" (meta-driver task)
  that takes care of this kind of stuff].

  Don't get too fancy here. As long as we don't have a stable driver
  API, it would not be very useful to try to port Linux drivers to
  this new architecture yet.

* Port a real (Linux? *BSD?) device driver to the DMGR architecture.
  We especially need the port to some popular ATA and SCSI drivers,
  both for the external pager's backing store (and later by
  filesystem/storeio translators to get files!!).

* [Other L4 hacking in a subsequent version of the TODO list].

Hurd-Area:
----------

* Convert the MIG *.defs to IDL, then generate the Mach stubs with
  Flick. Fix things that break here.

* Extract the Mach and Hurd sysdeps of glibc into a separate
  library. We need the Hurdish-code available outside glibc
  in non-glibc environments like naked L4 (or L4 with OSKit),
  but also in the VK/{non-glibc-guest-os,BSD,...} cases.

  Some may argue that it would be best to port glibc to L4
  altogether. This is IMHO not a feasible option right now.
  Why? Well, look at the glibc sources! glibc works best,
  if it can assume a POSIX like interface from it's underlying
  kernel (Linux' int 0x80 trap). In cases of a "weak"/slim
  kernel like L4, too much is missing and would need to be
  mapped in glibc's sysdeps. Just look at the combined
  mach/hurd sysdeps (together with Mach's powerful semantics
  and the current Hurd servers). In L4's case, the task of
  porting glibc would be even much harder than that.

  I'd just like to point out, that glibc currently supports
  _only_ Linux (besides the Hurd) as its backing kernel.
  Not seing a {Free,Open,Net}BSD port underway is IMO
  strong evidence that porting glibc won't be that easy,
  [considering that the *BSD's kernel APIs are pretty close
   to Linux, this is especially puzzling...].

  So, for practical bootstrapping reasons, we'll need to get
  the Hurd to L4 first, before we can even think of porting
  glibc itself. There is a chicken-and-egg problem here that
  is not really easy to solve. IMHO, the best way to get
  started, would be to get a nice clean libhurd.a that could
  be linked to the rest of the Hurd.

* Mach consumes and hands out port rights on IPC. This is heavily
  used in the Hurd. Add an indirection level here by writing a
  port rights server task, that would be IPC-d from clients.
  The pr-task would act as a proxy by managing the port rights
  itself (getting them from and sending them to Mach and userland
  tasks).

  Everything should go through this proxy, using a well-defined API.
  Of course, it will slow everything down on Mach (!!!), but it is
  the easiest way for a preliminary port to L4.

  The pr-task would mirror some of Mach's state (the port rights)
  in userland, and could as such be directly ported to an L4 task.
  In L4, pr-task would generate the port rights itself rather than
  getting them from the kernel.

  This is not an ideal solution, just an interim one [it's
  basically a lite version of Okuji's l4-mach task idea
  or Michael Hohmuth's and Sven Rudolph's Mach/L3 server,
  only much smaller!!!]

* Think about the necessary changes to IPC, so that synchroneous
  IPC won't break things. Under L4, only synchroneous IPC is
  allowed, wether under Mach, all IPC is asynchroneous. For the
  uninitiated, synchroneous IPC means that for an IPC to take
  place, the receiver must already have invoked a ipc_receive()
  (and the invoking thread will be blocked [unless a timeout is set])
  so that the sender's ipc_send() can succeed. Data is not buffered
  in the kernel, as in the asynchroneous Mach case.

  One area where this can become an issue is MIG, where the *.defs
  specify some service xxx.defs and xxx_reply.defs. With synch.
  IPC, request and receive are atomic. Is this an issue at all?

  Fix or change everything that depends on Mach buffering the
  messages. We need a synchroneous Hurd before we can port it ;)

* With the help of pr-task written earlier, design a simplified
  VK-API that is used by the Hurd and by glibc. Especially,
  replace mach_*() calls with calls to pr-task and to this
  simplified VK. You can safely assume that the vm_*() calls
  are part of the VK, because on the L4 side, there will be
  an external pager + library that will provide those.

  The vk_*() calls would simply be macros or thin wrappers around
  already existing Mach functions. They shouldn't break anything
  here, because it's just renaming in case of Hurd/Mach.

  Renaming every kernel-invoking function with vk_*() also has
  the advantage of providing a clear and succint "kernel-API"
  to the "kernel-layer" (see VK-Layer below).

  When porting to L4, the vk_*() functions will be reimplemented
  using L4-specific techniques and -servers/libraries.

VK-Area:
--------

* Read about the need for a microkernel neutral 'virtual kernel'
  layer for the Hurd (and other OSes) in the archives of l4-hurd.

* Look at the vk_*() functions generated above and identify what
  is mach-specific and what seems generic enough to be present
  in every [micro]kernel and run-time environment needed.

* Define a small (but not necessarily minimalistic) VK-API.
  Implement it on at least the following platforms:
    Mach, L4 and POSIX.

* Write some simple clients to this VK to see if everything
  works as expected. Refine the API and the implementation
  if necessary.

* Port the Hurd to this VK-API (call it pHurd for 'portable' Hurd).
  The final design would be:

     Layer 3: pHurd, other pOSes
     Layer 2: VK (VK/L4, VK/Mach, VK/POSIX)
     Layer 1: microkernel (L4, Mach) or monolithic kernel
     Layer 0: Hardware

Please contribute to the discussion here! I'm currently very busy
with a non L4/Hurd related project (sigh...) that I'm lacking time
at the moment to actively participate. But I'm willing to help
everyone to get started.

Thanks,

-Farid.

-- 
Farid Hajji -- Unix Systems and Network Admin | Phone: +49-2131-67-555
Broicherdorfstr. 83, D-41564 Kaarst, Germany  | address@hidden
- - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - -
One OS To Rule Them All And In The Darkness Bind Them... --Bill Gates.