support live migration in NVMe/vfio-user

[Thanos Makatos]

Changpeng, I've started looking at how to implement live migration for 
NVMe/vfio-user. Live migration in libvfio-user is implemented using the VFIO 
live migration protocol but this complexity is somewhat hidden to the user; we 
present a slightly simpler API. The server needs to implement a set of 
callbacks that handle migration device state (running, pre-copy, stop-and-copy, 
resuming) and another set of callbacks that allow the client to read/write 
device state (as it's the client responsible for forwarding device state to the 
destination). In 
https://github.com/nutanix/libvfio-user/blob/master/samples/client.c and 
https://github.com/nutanix/libvfio-user/blob/master/samples/server.c I've 
implemented a simple sample of how migration works in libvfio-user (it doesn't 
use a pre-copy scenario, only a stop-and-copy one).

Since VFIO live migration is not yet supported in mpqemu, I've been looking at 
implementing a utility to drive the migration (based on 
examples/nvme/identify/identify.c), I've put this code here: 
https://github.com/tmakatos/spdk/tree/migr (it contains lots of hacks, as usual 
;)). I got as far as the client setting the device state to stop-and-copy (the 
server asserts in the device state change callback). We need to start thinking 
what state we need to send to the server, how/when we can quiesce queues, how 
the new server resume from stored state, etc. Also, we should look at the NVMe 
spec to see whether there are mechanisms that could allow us to simplify the 
implementation.

Is there some specific state SPDK stores for a controller? I guess we could 
start with implementing the stop-and-copy phase.