Re: [PATCH] vhost-vsock: fix migration issue when seqpacket is supported

[Stefano Garzarella]

On Fri, Sep 10, 2021 at 02:35:53AM -0400, Michael S. Tsirkin wrote:
> On Thu, Sep 09, 2021 at 10:02:12AM +0100, Daniel P. Berrangé wrote:
> > On Thu, Sep 09, 2021 at 04:47:42AM -0400, Michael S. Tsirkin wrote:
> > > On Tue, Sep 07, 2021 at 02:22:24PM +0100, Daniel P. Berrangé wrote:
> > > > On Tue, Sep 07, 2021 at 02:49:35PM +0200, Stefano Garzarella wrote:
> > > > > Commit 1e08fd0a46 ("vhost-vsock: SOCK_SEQPACKET feature bit support")
> > > > > enabled the SEQPACKET feature bit.
> > > > > This commit is released with QEMU 6.1, so if we try to migrate a VM where
> > > > > the host kernel supports SEQPACKET but machine type version is less than
> > > > > 6.1, we get the following errors:
> > > > >
> > > > >     Features 0x130000002 unsupported. Allowed features: 0x179000000
> > > > >     Failed to load virtio-vhost_vsock:virtio
> > > > >     error while loading state for instance 0x0 of device 
> > '0000:00:05.0/virtio-vhost_vsock'
> > > > >     load of migration failed: Operation not permitted
> > > > >
> > > > > Let's disable the feature bit for machine types < 6.1, adding a
> > > > > `features` field to VHostVSock to simplify the handling of upcoming
> > > > > features we will support.
> > > >
> > > > IIUC, this will still leave migration broken for anyone migrating
> > > > a >= 6.1 machine type between a kernel that supports SEQPACKET and
> > > > a kernel lacking that, or vica-verca.
> > > >
> > > > If a feature is dependant on a host kernel feature we can't turn
> > > > that on automatically as part of the machine type, as we need
> > > > ABI stability across migration indepdant of kernel version.
> > > >
> > > >
> > > > Regards,
> > > > Daniel
> > >
> > > This is a fundamental problem we have with kernel accelerators.
> > > A higher level solution at management level is needed.
> > > For now yes, we do turn features on by default,
> > > consistent kernels on source and destination are assumed.
> > > For downstreams not a problem at all as they update
> > > userspace and kernel in concert.
> >
> > Even downstream in RHEL that is not actually valid anymore. Container
> > based deployment has killed any assumptions that can be made in this
> > respect. Even if the userspace and kernel are updated in lockstep in
> > a particular RHEL release, you cannot assume the running environment
> > will have a matched pair.
> >
> > Users can be running QEMU userspace from RHEL-8.5 inside a container
> > that has been deployed on a host using a 8.3 kernel. We've even had
> > cases of running QEMU from RHEL-8, on a RHEL-7 host.
> >
> > Regards,
> > Daniel
>
> Is there finally an interest in addressing this then?  This would
> involve collecting host features across a cluster and for each host
> figuring out a configuration that works for migration. IIRC a tool was
> proposed for the task (to live alongside e.g. qemu-img).

Apart from the tool, what if we provide a mechanism for adding/removing 
device features at run-time?
After migration we could tell the guest that a feature is no longer 
available.

Maybe it's too complicated, but it would allow us to solve the problem 
of migrating between different kernels or, with vDPA, between different 
devices that don't support all features.

> As long as we just stick to the machine type the best we can do is
> probably to keep doing what we do now (hope that the two host kernels
> are more or less consistent) as otherwise we'd have to never enable any
> new features in vsock.

Should we at least merge this patch to allow to migrate a VM between a 
new and an old qemu even if the kernel is the same?

Thanks,
Stefano