Re: [RFC PATCH 00/13] Add a plugin to support out-of-band live migration for VFIO pass-through device

[Alex Williamson]

On Tue, 24 May 2022 14:18:35 +0800
Lei Rao <lei.rao@intel.com> wrote:

> Migration of a VFIO passthrough device can be supported by using a device 
> specific kernel driver to save/restore the device state thru device specific 
> interfaces. But this approach doesn't work for devices that lack a state 
> migration interface, e.g. NVMe.
> 
> On the other hand, Infrastructure Process Unit (IPU) or Data Processing Unit 
> (DPU) vendors may choose to implement an out-of-band interface from the SoC 
> to 
> help manage the state of such non-migratable devices e.g. via gRPC or 
> JSON-RPC 
> protocols.
> 
> This RFC attempts to support such out-of-band migration interface by 
> introducing
> the concept of migration backends in vfio. The existing logic around vfio 
> migration uAPI is now called the 'local' backend while a new 'out-of-band' 
> backend is further introduced allowing vfio to redirect VMState ops to an 
> external plugin.
> 
> Currently, the backend migration Ops is defined close to SaveVMHandlers. We 
> also
> considered whether there is value of abstracting it in a lower level e.g. 
> close 
> to vfio migration uAPI but no clear conclusion. Hence this is one part which 
> we'd like to hear suggestions.
> 
> This proposal adopts a plugin mechanism (an example can be found in [1]) 
> given 
> that IPU/DPU vendors usually implement proprietary migration interfaces 
> without
> a standard. But we are also open if an alternative option makes better sense,
> e.g. via loadable modules (with Qemu supporting gRPC or JSON-RPC support) or 
> an
> IPC mechanism similar to vhost-user.

AFAIU, QEMU is not interested in supporting plugin modules, especially
proprietary ones.  I don't see that a case has really been made that
this cannot be done in-band, through a vfio-pci variant driver,
possibly making use of proprietary interfaces to a userspace agent if
necessary (though please don't ask such to be accepted in-tree for the
kernel either).  A vfio-user device server might also host such
proprietary, device specific agents while supporting the standard,
in-band migration interface.  Thanks,

Alex

> 
> The following graph describes the overall component relationship:
> 
>  +----------------------------------------------------+
>  | QEMU                                               |
>  | +------------------------------------------------+ |
>  | |        VFIO Live Migration Framework           | |
>  | |    +--------------------------------------+    | |
>  | |    |         VFIOMigrationOps             |    | |
>  | |    +-------^---------------------^--------+    | |
>  | |            |                     |             | |
>  | |    +-------v-------+     +-------v--------+    | |
>  | |    | LM Backend Via|     | LM Backend Via |    | |
>  | |    |   Device Fd   |     |    Plugins     |    | |
>  | |    +-------^-------+     |     +----------+    | |
>  | |            |             |     |Plugin Ops+----+-+------------+
>  | |            |             +-----+----------+    | |            |
>  | |            |                                   | |  
> +---------v----------+
>  | +------------+-----------------------------------+ |  |  Vendor Specific   
> |
>  |              |                                     |  |    Plugins(.so)    
> |
>  +--------------+-------------------------------------+  
> +----------+---------+
>   UserSpace     |                                                   |
> ----------------+---------------------------------------------      |
>   Kernel        |                                                   |
>                 |                                                   |
>      +----------v----------------------+                            |
>      |        Kernel VFIO Driver       |                            |
>      |    +-------------------------+  |                            |
>      |    |                         |  |                            | Network
>      |    | Vendor-Specific Driver  |  |                            |
>      |    |                         |  |                            |
>      |    +----------^--------------+  |                            |
>      |               |                 |                            |
>      +---------------+-----------------+                            |
>                      |                                              |
>                      |                                              |
> ---------------------+-----------------------------------------     |
>   Hardware           |                                              |
>                      |            +-----+-----+-----+----+-----+    |
>           +----------v------+     | VF0 | VF1 | VF2 | ...| VFn |    |
>           |   Traditional   |     +-----+-----+-----+----+-----+    |
>           |  PCIe Devices   |     |                            |    |
>           +-----------------+     |   +--------+------------+  |    |
>                                   |   |        |   Agent    |<-+----+
>                                   |   |        +------------+  |
>                                   |   |                     |  |
>                                   |   | SOC                 |  |
>                                   |   +---------------------+  |
>                                   | IPU                        |
>                                   +----------------------------+
> 
> Two command-line parameters (x-plugin-path and x-plugin-arg) are introduced 
> to 
> enable the out-of-band backend. If specified, vfio will attempt to use the 
> out-of-band backend.
> 
> The following is an example of VFIO command-line parameters for OOB-Approach:
> 
>   -device 
> vfio-pci,id=$ID,host=$bdf,x-enable-migration,x-plugin-path=$plugin_path,x-plugin-arg=$plugin_arg
> 
> [1] https://github.com/raolei-intel/vfio-lm-plugin-example.git
> 
> Lei Rao (13):
>   vfio/migration: put together checks of migration initialization
>     conditions
>   vfio/migration: move migration struct allocation out of
>     vfio_migration_init
>   vfio/migration: move vfio_get_dev_region_info out of
>     vfio_migration_probe
>   vfio/migration: Separated functions that relate to the In-Band
>     approach
>   vfio/migration: rename functions that relate to the In-Band approach
>   vfio/migration: introduce VFIOMigrationOps layer in VFIO live
>     migration framework
>   vfio/migration: move the statistics of bytes_transferred to generic
>     VFIO migration layer
>   vfio/migration: split migration handler registering from
>     vfio_migration_init
>   vfio/migration: move the functions of In-Band approach to a new file
>   vfio/pci: introduce command-line parameters to specify migration
>     method
>   vfio/migration: add a plugin layer to support out-of-band live
>     migration
>   vfio/migration: add some trace-events for vfio migration plugin
>   vfio/migration: make the region and plugin member of struct
>     VFIOMigration to be a union
> 
>  docs/devel/vfio-migration-plugin.rst    | 165 +++++++
>  hw/vfio/meson.build                     |   2 +
>  hw/vfio/migration-local.c               | 456 +++++++++++++++++++
>  hw/vfio/migration-plugin.c              | 266 +++++++++++
>  hw/vfio/migration.c                     | 577 ++++++------------------
>  hw/vfio/pci.c                           |   2 +
>  hw/vfio/trace-events                    |   9 +-
>  include/hw/vfio/vfio-common.h           |  37 +-
>  include/hw/vfio/vfio-migration-plugin.h |  21 +
>  9 files changed, 1096 insertions(+), 439 deletions(-)
>  create mode 100644 docs/devel/vfio-migration-plugin.rst
>  create mode 100644 hw/vfio/migration-local.c
>  create mode 100644 hw/vfio/migration-plugin.c
>  create mode 100644 include/hw/vfio/vfio-migration-plugin.h
>