Re: Why is it so damn slow ?

[Richard Braun]

On Sun, Dec 02, 2012 at 11:15:08AM +0100, Richard Braun wrote:
> On Sat, Dec 01, 2012 at 12:53:30PM +0100, Thomas Schwinge wrote:
> > So, switching to a SSD does give a small performance boost, but not the
> > big one we had hoped for.  Using --no-atime improves this further.
> > Perhaps we should follow suit with the Linux kernel, and implement and
> > make default a »relatime« mount option.
> 
> Yes, we should probably make relatime the default. Concerning the much
> less than expected performance boost, I don't consider it surprising.
> Last summer (while working on the VM cache, select with a timeout and
> the port to pthreads), I noticed that much of the time spent by the CPU
> is actually wasted for two major reasons.
> 
> One is the way the VM cache works, i.e. caching memory objects instead
> of their backing pages, with a fixed limit (Mach can actually have
> cached objects with no single page in memory !), but I think this reason
> is now well known. The other is less obvious : the Hurd uses a lot of
> spin locks, some global and heavily used such as _ports_lock in
> libports. These spin locks are a plague because, once the number of
> threads is significant (a number that varies with the raw performance of
> the hardware), they will make servers spend much time checking the lock
> state and invoking the scheduler, which is then unable to make any more
> good decisions because all those threads are running. Worse, before
> switching to another thread, the caller depresses its priority, so in
> the end, there can be cases like the one you reported a few days ago, of
> servers that, at some point (under load), have hundreds or even
> thousands of threads spinning with a very low priority. Let's not even
> mention fairness.

After some testing, with a patch that replaces almost all spin locks
with mutexes, it turns out that the real problem is actually contention,
whatever the lock type. The mutex version would "fail" (that is, the
system would become unresponsive) a bit later than the spin lock one,
but there are simply too many threads after some point to allow the
system to progress, and with additional I/O, this number doesn't stop
increasing until another failure (not being able to create new threads)
occur.

This contention also seems to be the reason why using a fast storage
device doesn't significantly help performance.

-- 
Richard Braun