Re: [PATCH] vhost-vsock: fix migration issue when seqpacket is supported

[Michael S. Tsirkin]

On Mon, Sep 13, 2021 at 02:51:42PM +0200, Stefano Garzarella wrote:
> On Fri, Sep 10, 2021 at 02:35:53AM -0400, Michael S. Tsirkin wrote:
> > On Thu, Sep 09, 2021 at 10:02:12AM +0100, Daniel P. Berrangé wrote:
> > > On Thu, Sep 09, 2021 at 04:47:42AM -0400, Michael S. Tsirkin wrote:
> > > > On Tue, Sep 07, 2021 at 02:22:24PM +0100, Daniel P. Berrangé wrote:
> > > > > On Tue, Sep 07, 2021 at 02:49:35PM +0200, Stefano Garzarella wrote:
> > > > > > Commit 1e08fd0a46 ("vhost-vsock: SOCK_SEQPACKET feature bit 
> > > > > > support")
> > > > > > enabled the SEQPACKET feature bit.
> > > > > > This commit is released with QEMU 6.1, so if we try to migrate a VM 
> > > > > > where
> > > > > > the host kernel supports SEQPACKET but machine type version is less 
> > > > > > than
> > > > > > 6.1, we get the following errors:
> > > > > >
> > > > > >     Features 0x130000002 unsupported. Allowed features: 0x179000000
> > > > > >     Failed to load virtio-vhost_vsock:virtio
> > > > > >     error while loading state for instance 0x0 of device 
> > > > > > '0000:00:05.0/virtio-vhost_vsock'
> > > > > >     load of migration failed: Operation not permitted
> > > > > >
> > > > > > Let's disable the feature bit for machine types < 6.1, adding a
> > > > > > `features` field to VHostVSock to simplify the handling of upcoming
> > > > > > features we will support.
> > > > >
> > > > > IIUC, this will still leave migration broken for anyone migrating
> > > > > a >= 6.1 machine type between a kernel that supports SEQPACKET and
> > > > > a kernel lacking that, or vica-verca.
> > > > >
> > > > > If a feature is dependant on a host kernel feature we can't turn
> > > > > that on automatically as part of the machine type, as we need
> > > > > ABI stability across migration indepdant of kernel version.
> > > > >
> > > > >
> > > > > Regards,
> > > > > Daniel
> > > >
> > > > This is a fundamental problem we have with kernel accelerators.
> > > > A higher level solution at management level is needed.
> > > > For now yes, we do turn features on by default,
> > > > consistent kernels on source and destination are assumed.
> > > > For downstreams not a problem at all as they update
> > > > userspace and kernel in concert.
> > > 
> > > Even downstream in RHEL that is not actually valid anymore. Container
> > > based deployment has killed any assumptions that can be made in this
> > > respect. Even if the userspace and kernel are updated in lockstep in
> > > a particular RHEL release, you cannot assume the running environment
> > > will have a matched pair.
> > > 
> > > Users can be running QEMU userspace from RHEL-8.5 inside a container
> > > that has been deployed on a host using a 8.3 kernel. We've even had
> > > cases of running QEMU from RHEL-8, on a RHEL-7 host.
> > > 
> > > Regards,
> > > Daniel
> > 
> > Is there finally an interest in addressing this then?  This would
> > involve collecting host features across a cluster and for each host
> > figuring out a configuration that works for migration. IIRC a tool was
> > proposed for the task (to live alongside e.g. qemu-img).
> 
> Apart from the tool, what if we provide a mechanism for adding/removing
> device features at run-time?
> After migration we could tell the guest that a feature is no longer
> available.
> 
> Maybe it's too complicated, but it would allow us to solve the problem of
> migrating between different kernels or, with vDPA, between different devices
> that don't support all features.

Possible going forward but not supported by the spec at this point,
and tricky to do generally.
It's possible to do it in a vsock specific way since sockets
are currently closed across migration.


> > 
> > As long as we just stick to the machine type the best we can do is
> > probably to keep doing what we do now (hope that the two host kernels
> > are more or less consistent) as otherwise we'd have to never enable any
> > new features in vsock.
> 
> Should we at least merge this patch to allow to migrate a VM between a new
> and an old qemu even if the kernel is the same?
> 
> Thanks,
> Stefano

I'm inclined to do this, yes.

-- 
MST