The workings of the pthreads ufs patch

[Thomas DiModica]

I'm CCing this to bug-hurd.

So, this is the long version of what my pthreads patch to ufs does.
If anyone finds my beliefs to be in error, please speak up.

I will begin by describing how a cthread rwlock worked, in brief. Inside the
cthread rwlock, there is a condition that all waiters sleep on. When a writer
or the last reader leaves, they signal this condition, waking up all waiters
to see if they can acquire the lock.

In ufs, we have an rwlock that protects a structure. That structure manages a
resource. Threads that want to modify the structure have to take the lock for
writing. Threads that want to read from or write to the resource take the lock
for reading. As an optimization, at special times, specifically during the
resizing of the resource, threads that only want to read from the resource are
allowed to do so without taking the lock at all.

In pager.c, find_address, we have the ugly mutated inlining of
rwlock_reader_lock. This was to allow that optimization. Readers waiting to
read from the resource don't try to take the lock if the condition that they
are sleeping on is signaled: they first go back to see if they need to take
the lock at all. This works with sizes.c, block_extended, where the condition
gets broadcast. Here, after setting the flag to allow unlocked access, we
would reach into the rwlock and signal its condition.

In pthreads, rwlocks are implemented with wait queues instead of a condition.
Even if OUR implementation of pthreads used a condition, we would want to
remove this for the sake of portability. This is why I added a condition
variable: to replace the condition variable inside the rwlock, so that readers
waiting to take the lock to read from the resource could be awakened.
Hopefully, my description of what was going on will make what I've done make
more sense.

The only problem with this setup is that every time the lock is taken for
writing, the writer has to explicitly broadcast the condition to wake any
readers that were waiting for the lock.

Richard, if you wish to remove this optimization for the sake of clarity, be
my guest. I was just trying to be faithful to the working of the original
code.

Thomas D