Thread model (was: Ext2 superblock fault)

[olafBuddenhagen]

Hi,

On Sun, Mar 09, 2008 at 10:17:05PM -0400, Thomas Bushnell BSG wrote:
> On Mon, 2008-03-10 at 01:19 +0000, Samuel Thibault wrote:

> > This thread is syncing everything, i.e. asking a lot of writes,
> > which triggers the creation of a lot of threads.  Unfortunately the
> > superblock was paged out, so they all block on reading it.
> > Unfortunately, since in Debian there is a patch which limits the
> > number of created threads, the read of the superblock doesn't
> > actually create a new thread, that is delayed.  But since none of
> > the existing threads can progress (since they are all waiting for
> > the super block), things are just dead locked...
> 
> As a general rule, the Hurd always assumes that an RPC can be handled;
> this is quite emebedded in the way diskfs works.
> 
> A patch which limits the number of threads is inherently buggy in the
> Hurd, and that patch MUST be disabled for anything to work properly.

I'm glad this discussion came up at last: This is a very serious issue,
and your input is necessary.

The real problem here is that the current thread model of the Hurd
servers is fundamentally broken. Creating a new kernel thread for each
incoming RPC just doesn't work. The amount of incoming RPCs can be
pretty much unlimited, but kernel threads are a limited resource -- a
quite expensive resource, in fact.

And this is not a theoretical problem, but a very real one: On heavy
disk load, the filesystem server easily created hundreds, even thousands
of threads. During large compile jobs for example the Hurd was crashing
regularily with zalloc panics. And the problem became more and more
pressing with machines getting faster and disk space usage larger.

Another easy way to reproduce the problem is to create a process with
many children, each opening /dev/null many times (the number of open
files per process is limited, thus we need many children), and then
killing all the children quickly (e.g. by terminating the parent). With
some 30000 or so total open ports, this is guaranteed to result in a
zalloc panic -- the null server is hammered with dead name
notifications, resulting again in thread explosion.

When these issues became known, Sergio Lopez implemented a hack that
simply limits the number of thread created by each single server to a
fixed amount. When more RPCs come in, they won't be handled until some
of the threads becomes free.

While this patch did wonders for stability under load, and no problems
showed at first, it immediately struck me as having potential for
causing deadlocks. When I pointed that out, Sergio made a slight
modification: Rather than completely disabling creation of new threads
once the limit is reached, each further thread becomes active only after
idling for two seconds -- so additional threads get created very slowly.

I never was happy with that solution(?), and suggested a more adaptive
approach: Keep track of the existing threads, and if none of them makes
progress in a certain amount of time (say 100 ms), allow creating some
more threads. But that was never implemented. Also, it still might cause
considerable delays in some situations; and I'm not even sure it would
fix all problems. (I didn't fully understand the problem discussed in
this thread, so I don't know whether it would be fixed by that?)

And anyways, it still would be just an ugly workaround. The real
solution here of course is to fix the thread model -- using some kind of
continuation mechanism: Have a limited number of threads (ideally one
per CPU) handle incoming requests. Whenever some operation would require
blocking for some event (in the case of diskfs, waiting for the
underlying store to finish reading/writing), the state is instead saved
to some list of outstanding operations, and the thread goes on handling
other requests. Only when the event completes, we read the state back
and continue handling the original request.

Of course, that will be a major change; it requires modification of
considerable parts of the Hurd servers. But it seems the only way to
handle this properly. What do you think?

I wonder whether I should add this to the list of project ideas for
GSoC...

-antrik-