RE: [question] VFIO Device Migration: The vCPU may be paused during vfio device DMA in iommu nested stage mode && vSVA

[Tian, Kevin]

> From: Kunkun Jiang <jiangkunkun@huawei.com>
> Sent: Monday, September 27, 2021 8:30 PM
> 
> Hi Kevin:
> 
> On 2021/9/24 14:47, Tian, Kevin wrote:
> >> From: Kunkun Jiang <jiangkunkun@huawei.com>
> >> Sent: Friday, September 24, 2021 2:19 PM
> >>
> >> Hi all,
> >>
> >> I encountered a problem in vfio device migration test. The
> >> vCPU may be paused during vfio-pci DMA in iommu nested
> >> stage mode && vSVA. This may lead to migration fail and
> >> other problems related to device hardware and driver
> >> implementation.
> >>
> >> It may be a bit early to discuss this issue, after all, the iommu
> >> nested stage mode and vSVA are not yet mature. But judging
> >> from the current implementation, we will definitely encounter
> >> this problem in the future.
> > Yes, this is a known limitation to support migration with vSVA.
> >
> >> This is the current process of vSVA processing translation fault
> >> in iommu nested stage mode (take SMMU as an example):
> >>
> >> guest os            4.handle translation fault 5.send CMD_RESUME to vSMMU
> >>
> >>
> >> qemu                3.inject fault into guest os 6.deliver response to
> >> host os
> >> (vfio/vsmmu)
> >>
> >>
> >> host os              2.notify the qemu 7.send CMD_RESUME to SMMU
> >> (vfio/smmu)
> >>
> >>
> >> SMMU              1.address translation fault              8.retry or
> >> terminate
> >>
> >> The order is 1--->8.
> >>
> >> Currently, qemu may pause vCPU at any step. It is possible to
> >> pause vCPU at step 1-5, that is, in a DMA. This may lead to
> >> migration fail and other problems related to device hardware
> >> and driver implementation. For example, the device status
> >> cannot be changed from RUNNING && SAVING to SAVING,
> >> because the device DMA is not over.
> >>
> >> As far as i can see, vCPU should not be paused during a device
> >> IO process, such as DMA. However, currently live migration
> >> does not pay attention to the state of vfio device when pausing
> >> the vCPU. And if the vCPU is not paused, the vfio device is
> >> always running. This looks like a *deadlock*.
> > Basically this requires:
> >
> > 1) stopping vCPU after stopping device (could selectively enable
> > this sequence for vSVA);
> How to tell if vSVA is open?
> In fact, as long as it is in iommu nested stage mode, there will
> be such a problem, whether it is vSVA or no-vSVA. In no-vSVA mode,
> a fault can also be generated by modifying the guest device driver.

You don't need tell if vSVA is open. The kernel driver knows
whether nested is enabled and whether fault is handled by 
userspace. with this knowledge the driver can generally indicate 
such order restriction to Qemu via migration region info.

> >
> > 2) when stopping device, the driver should block new requests
> > from vCPU (queued to a pending list) and then drain all in-fly
> > requests including faults;
> >      * to block this further requires switching from fast-path to
> > slow trap-emulation path for the cmd portal before stopping
> > the device;
> >
> > 3) save the pending requests in the vm image and replay them
> > after the vm is resumed;
> >      * finally disable blocking by switching back to the fast-path for
> > the cmd portal;
> Is there any related patch sent out and discussed? I might have
> overlooked that.

Not yet. As you said even vSVA support is not in place. It's too 
early to send patches for other features on top of that.

Thanks,
Kevin