Re: [PATCH] vhost-vsock: fix migration issue when seqpacket is supported

[Michael S. Tsirkin]

On Tue, Sep 14, 2021 at 12:42:09PM +0200, Stefano Garzarella wrote:
> On Mon, Sep 13, 2021 at 09:46:48AM -0400, Michael S. Tsirkin wrote:
> > On Mon, Sep 13, 2021 at 02:51:42PM +0200, Stefano Garzarella wrote:
> > > On Fri, Sep 10, 2021 at 02:35:53AM -0400, Michael S. Tsirkin wrote:
> > > > On Thu, Sep 09, 2021 at 10:02:12AM +0100, Daniel P. Berrangé wrote:
> > > > > On Thu, Sep 09, 2021 at 04:47:42AM -0400, Michael S. Tsirkin wrote:
> > > > > > On Tue, Sep 07, 2021 at 02:22:24PM +0100, Daniel P. Berrangé wrote:
> > > > > > > On Tue, Sep 07, 2021 at 02:49:35PM +0200, Stefano Garzarella 
> > > > > > > wrote:
> > > > > > > > Commit 1e08fd0a46 ("vhost-vsock: SOCK_SEQPACKET feature bit 
> > > > > > > > support")
> > > > > > > > enabled the SEQPACKET feature bit.
> > > > > > > > This commit is released with QEMU 6.1, so if we try to migrate 
> > > > > > > > a VM where
> > > > > > > > the host kernel supports SEQPACKET but machine type version is 
> > > > > > > > less than
> > > > > > > > 6.1, we get the following errors:
> > > > > > > >
> > > > > > > >     Features 0x130000002 unsupported. Allowed features: 
> > > > > > > > 0x179000000
> > > > > > > >     Failed to load virtio-vhost_vsock:virtio
> > > > > > > >     error while loading state for instance 0x0 of device 
> > > > > > > > '0000:00:05.0/virtio-vhost_vsock'
> > > > > > > >     load of migration failed: Operation not permitted
> > > > > > > >
> > > > > > > > Let's disable the feature bit for machine types < 6.1, adding a
> > > > > > > > `features` field to VHostVSock to simplify the handling of 
> > > > > > > > upcoming
> > > > > > > > features we will support.
> > > > > > >
> > > > > > > IIUC, this will still leave migration broken for anyone migrating
> > > > > > > a >= 6.1 machine type between a kernel that supports SEQPACKET and
> > > > > > > a kernel lacking that, or vica-verca.
> > > > > > >
> > > > > > > If a feature is dependant on a host kernel feature we can't turn
> > > > > > > that on automatically as part of the machine type, as we need
> > > > > > > ABI stability across migration indepdant of kernel version.
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > > Daniel
> > > > > >
> > > > > > This is a fundamental problem we have with kernel accelerators.
> > > > > > A higher level solution at management level is needed.
> > > > > > For now yes, we do turn features on by default,
> > > > > > consistent kernels on source and destination are assumed.
> > > > > > For downstreams not a problem at all as they update
> > > > > > userspace and kernel in concert.
> > > > >
> > > > > Even downstream in RHEL that is not actually valid anymore. Container
> > > > > based deployment has killed any assumptions that can be made in this
> > > > > respect. Even if the userspace and kernel are updated in lockstep in
> > > > > a particular RHEL release, you cannot assume the running environment
> > > > > will have a matched pair.
> > > > >
> > > > > Users can be running QEMU userspace from RHEL-8.5 inside a container
> > > > > that has been deployed on a host using a 8.3 kernel. We've even had
> > > > > cases of running QEMU from RHEL-8, on a RHEL-7 host.
> > > > >
> > > > > Regards,
> > > > > Daniel
> > > >
> > > > Is there finally an interest in addressing this then?  This would
> > > > involve collecting host features across a cluster and for each host
> > > > figuring out a configuration that works for migration. IIRC a tool was
> > > > proposed for the task (to live alongside e.g. qemu-img).
> > > 
> > > Apart from the tool, what if we provide a mechanism for adding/removing
> > > device features at run-time?
> > > After migration we could tell the guest that a feature is no longer
> > > available.
> > > 
> > > Maybe it's too complicated, but it would allow us to solve the problem of
> > > migrating between different kernels or, with vDPA, between different 
> > > devices
> > > that don't support all features.
> > 
> > Possible going forward but not supported by the spec at this point,
> > and tricky to do generally.
> > It's possible to do it in a vsock specific way since sockets
> > are currently closed across migration.
> 
> Yep, I see.
> 
> > 
> > 
> > > >
> > > > As long as we just stick to the machine type the best we can do is
> > > > probably to keep doing what we do now (hope that the two host kernels
> > > > are more or less consistent) as otherwise we'd have to never enable any
> > > > new features in vsock.
> > > 
> > > Should we at least merge this patch to allow to migrate a VM between a new
> > > and an old qemu even if the kernel is the same?
> > > 
> > > Thanks,
> > > Stefano
> > 
> > I'm inclined to do this, yes.
> > 
> 
> If you haven't queued it yet, I'd like to send a v2 using an `on,off,auto`
> property: `auto` would be the actual behavior when enabled, `on` instead
> requires that the kernel supports the feature otherwise there is an error.
> 
> Thanks,
> Stefano

go ahead pls.