Re: [PATCH 8/8] rpctrace: use condition variable to keep messages in sequence

[Brent W. Baccala]

On Sat, Oct 29, 2016 at 1:48 PM, Brent W. Baccala <cosine@freesoft.org> wrote:

Yes, patch7 without patch8 can reorder messages as they're being resent, but patch8 uses the mutex introduced in patch7, so the ordering can't be reversed.

They could be combined into a single patch.  Do you want me to prepare a combined patch and email it in?

OK, I'm attaching the combined patch7+patch8.

I've studied it a bit more, and it has a race condition that looks unavoidable to me.

rpctrace on ext2fs exposes the problem.  ext2fs maps a region of memory that it manages itself (using libpager), then calls vm_copy() on that region, using its own task port.  This causes the kernel to make a memory_object_data_request back to ext2fs, which makes some more requests (mach_port_deallocate and vm_allocate) on the task port before replying with a memory_object_data_supply, which allows the original vm_copy to return.

So, it's calling vm_copy on its task port, and needs to do some more operations on the task port before the vm_copy completes.  And just like the problem I described two months ago ("mach_msg blocking on call to vm_map"), the mach_msg call sending the vm_copy will block even if you're not waiting for the reply message.

So... in rpctrace, I want to signal the condition variable (allowing the next message to be processed) after the message has been sent, but since mach_msg blocks, I can't do that without blocking the entire task port, which deadlocks the process.

My solution is to signal the condition variable right before the call to mach_msg, but this creates a race condition where messages can get reordered.

As you know, I never liked this blocking behavior of mach_msg, but I just can't see any way around it now.  If you can suggest something, let me know...

    agape

    brent

0001-multithread-rpctrace-to-avoid-deadlocks-in-the-kerne.patch
Description: Text Data