This series of patches aims to minimize the downtime during live migration of a
virtio-net device with a vhost-user backend. In the case of hardware virtual
Data Path Acceleration (vDPA) implementation, the hardware configuration, which
includes tasks like VQ creation and RSS setting, may take above 200ms. This
significantly increases the downtime of the VM, particularly in terms of
networking.
To reduce the VM downtime, the proposed approach involves capturing the basic
device state/configuration during the VM's running stage and performing the
initial device configuration(presetup). During the normal configuration process
when the VM is in a stopped state, the second configuration is compared to the
first one, and only the differences are applied to reduce downtime. Ideally,
only the vring available index needs to be changed within VM stop.
This feature is disabled by default, because backend like dpdk also needs
adding support for vhost new message. New device property "x-early-migration"
can enable this feature.
1. Register a new vmstate for virtio-net with an early_setup flag to send the
device state during migration setup.
2. After device state load on destination VM, need to send device status to
vhost backend in a new way. Introduce new vhost-user message:
VHOST_USER_PRESETUP, to notify backend of presetup.
3. Let virtio-net, vhost-net, vhost-dev support presetup. Main flow:
a. vhost-dev sending presetup start.
b. virtio-net setting mtu.
c. vhost-dev sending vring configuration and setting dummy call/kick fd.
d. vhost-net sending vring enable.
e. vhost-dev sending presetup end.
TODOs:
======
- No vhost-vdpa/kernel support. Need to discuss/design new kernel interface
if there's same requirement for vhost-vdpa.
- No vIOMMU support so far. If there is a need for vIOMMU support, it is
planned to be addressed in a follow-up patchset.
Test:
=====
- Live migration VM with 2 virtio-net devices, ping can recover.
Together with DPDK patch [1].
- The time consumption of DPDK function dev_conf is reduced from 191.4 ms
to 6.6 ms.
References:
===========
[1] https://github.com/Mellanox/dpdk-vhost-vfe/pull/37
Any comments or feedback are highly appreciated.
Thanks,
Yajun
Yajun Wu (5):
vhost-user: Add presetup protocol feature and op
vhost: Add support for presetup
vhost-net: Add support for presetup
virtio: Add VMState for early load
virtio-net: Introduce LM early load
docs/interop/vhost-user.rst | 10 ++
hw/net/trace-events | 1 +
hw/net/vhost_net.c | 40 +++++++
hw/net/virtio-net.c | 100 ++++++++++++++++++
hw/virtio/vhost-user.c | 30 ++++++
hw/virtio/vhost.c | 166 +++++++++++++++++++++++++-----
hw/virtio/virtio.c | 152 ++++++++++++++++-----------
include/hw/virtio/vhost-backend.h | 3 +
include/hw/virtio/vhost.h | 12 +++
include/hw/virtio/virtio-net.h | 1 +
include/hw/virtio/virtio.h | 10 +-
include/net/vhost_net.h | 3 +
12 files changed, 443 insertions(+), 85 deletions(-)