Re: deferred cancellation of ipc

[Niels Möller]

Marcus Brinkmann <address@hidden> writes:

> The real problem is that the above of course doesn't work.  There is a race
> condition.  If the thread is "canceled" right after testcancel() and before
> the L4_Ipc(), the cancel flag is ignored for this Ipc, which might be a long
> blocking one.  This race window can possibly be quite small, but it is
> definitely there, and sticks out like a sore thumb.

Definitely sounds tricky. What is needed is a way to atomically check
the cancelled flag, and invoke the ipc system call.

Can one use the exchange registers to fake a function call in the
target thread? It should be possible to change the IP I guess, but
perhaps harder to save the return address.

Perhaps something like this could work?

  /* All of these should be thread local. */
  volatile int cancelled = 0;
  volatile int in_hurd_ipc = 0;
  volatile jmpbuf cancelbuf;
  
  hurd_ipc()
  {
    if (setjmp(&cancelbuf) != 0)
      {
      cancel;
        in_hurd_ipc = 0;
        return E_CANCEL;
      }
  
    /* Now the jmpbuf is valid. Tell the cancel hook about it. */
    in_hurd_ipc = 1;
  
    if (cancelled)
      goto cancel;
  
    ret = l4_ipc(...);
    in_hurd_ipc = 0;  /* X */
  
    return ret;
  }
  
  /* Always "called" by ExchangeRegisters munging if IP */
  hurd_cancel_hook()
  {
    cancelled = 1;
  
    if (in_hurd_ipc)
      longjmp(cancelbuf, 17);
    else
      /* Return and do nothing. May need something
         more complicated than return, if we can't get
         our return address on stack. */
      return;
  }

The idea is that to cancel the thread, one uses ExchangeRegisters to
cancel any current ipc operation, and change the IP to make the thread
execute the hurd_cancel_hook. This function sets the cancelled flag,
and if the thread is executing in the body of hurd_ipc, it longjmps
out.

The point is that the rase condition should be taken care of by
hurd_ipc *first* setting in_hurd_ipc, and *then* checking the
cancelled flag, and the oppisite order in hurd_cancel_hook.

A different race remains: If the thread is cancelled after the l4_ipc
system call has returned to user space, but before the line marked X
above, the returned value from the ipc is lost and the caller will
think it was cancelled. What we need here is for in_hurd_ipc to
cleared atomically, as the l4_ipc system call remains; almost anything
that's tied to thread ipc operations could work; a flag that's cleared
on ipc completion, or a counter of completed ipc, ...

Ahh, one idea just struck me: When cancelling a thread, install a
redirector for the thread. Then you can force all send ipc:s to fail
quickly, but it doesn't help for receive rpc:s, so perhaps you
considered that already. One nice thing is that the "forgot about
successful ipc" race is less serious for recv operations.

Perhaps one should handle cancel of "client" and "server" rpc:s using
different mechanisms: Install a redirector to handle the send case, and
use the setjmp hack for the receive case.

/Niels