Console deadlock & libports timeouts

[Diego Nieto Cid]

Hi,

Arch has been suffering from console lock-ups due to inactivity. The
system remains active and it's usable through other means such as a SSH
session.

On the other hand, Debian has showed no such behavior. The different
behaviors boil down to Debian's libports_stability[1] patch which is
missing in Arch.

Nonetheless, I'd like to document what's going on here.

The Hurd console's implementation is broken into two pieces each running
on it's own process, the console client and server.

The console server is also splitted into modules (input driver, display
driver, speaker,...) but they all run in the same process.

The console server puts itself as a translator on top of /dev/vcs folder
presenting the following hierarchy:
  
   + /dev/vcs
    \ 
     +- 1
         \
          +- console
          +- input
          +- display
     +- ..
     +- n

where the numbered nodes represent virtual consoles and it's contents
are all alike.

As the following graph shows, the console, input and display nodes are
the interfaces used by the terminal server, input driver and display
drivers respectively.

   +------------------+                                 +-----------------+
   |   Input driver   |                                 | Terminal Server |
   |                  |                                 |                 |
   |      pc-kbd      |                                 |                 |
   +------------------+                                 +-----------------+
            | _cons_vcons_input                                 |
            | writes to                                   reads |
            | vcs/i/input                         vcs/i/console |
            |                +-----------------+                |
            |                | Console Server  |                |
            |                |  /hurd/console  |                |
            |  input_enqueue | --------------- | input_dequeue  |
            +--------------->|   Input Queue   |>---------------+
                             | --------------- |
            +--------------->|  Output Buffer  |>---------------+
            |                +-----------------+                |
            |                                                   |
            | writes                                      reads |
            | vcs/i/console                       vcs/i/display |
            |                                                   |
   +----------------+                                   +-----------------+
   | Teminal Server |                                   | Display driver  |
   |                |                                   |                 |
   |   /hurd/term   |                                   |      vga        |
   +----------------+                                   +-----------------+


The input driver takes scancodes from the in-kernel kbd queue,
translates them into characters and writes them to the input node. Then
the terminal server reads the console node taking the characters out of
the console server.

Each of theese actions is actually an RPC handled by the translator on
/dev/vcs. Writes to input nodes are handled by calling input_enqueue[2] to
put the character into a queue. And reads from console nodes are handled
by calling input_dequeue[3] which takes out charecters from the queue
and gives them to the reader.

It's important to note here that both input_enqueue and input_dequeue
are blocking operations and a blocked input_dequeue necesarily needs an
input_enqueue call to continue.

Now back to why the console enters a dead lock state. RPCs are handled
by the console server with the help of libports'
ports_manage_multithreaded API.

When a message is receaved, the thread acting as receiver checks if any
other thread is also waiting for requests. If there is none a new thread
is spawned. Thus, the current thread continues processing the message
while the newly created thread starts listening for new ones.

Also, there are configurable timeouts for translators who want to go
away when they are not used. Apparently, this is where things get
buggy[4].

I don't know what problems are refered to in the cited email. But for
the console this means lack of request handlers. As showed by the
following test case:

  1. The console server's main thread is waiting for requests.
  2. One of the console nodes is read. The main thread spawns a request
     handler and then blocks when calling input_dequeue to satisfy the
     request.
  3. The request handler thread times out and goes away. But the
     translator remains alive as there's another thread doing it's work.
  4. A key is pressed causing a character to be written to an input
     node.

Now the system is locked waiting for an input_enqueue call which will
never happen.

Although I haven't tested it I beleive the patch sent as a reply should
fix this problem. Could someone review it? I'll try to test it as soon
as possible but I have to recompile hurd first.


[1] 
http://patch-tracker.debian.org/patch/series/view/hurd/20110519-3/libports_stability.patch
[2] http://git.savannah.gnu.org/cgit/hurd/hurd.git/tree/console/console.c#n1149
[3] http://git.savannah.gnu.org/cgit/hurd/hurd.git/tree/console/console.c#n1105
[4] http://lists.gnu.org/archive/html/bug-hurd/2001-08/msg00190.html


Regards,
--
Diego