RE: About restoring the state in vhost-vdpa device

[Parav Pandit]

Hi Gautam,

Please fix your email client to have right response format.
Otherwise, it will be confusing for the rest and us to follow the conversation.

More below.

> From: Gautam Dawar <gdawar@xilinx.com>
> Sent: Friday, May 13, 2022 1:48 PM

> > Our proposal diverge in step 7: Instead of enabling *all* the
> > virtqueues, only enable the CVQ.
> Just to double check, VQ 0 and 1 of the net are also not enabled, correct?
> [GD>>] Yes, that's my understanding as well.
> 
> > After that, send the DRIVER_OK and queue all the control commands to
> > restore the device status (MQ, RSS, ...). Once all of them have been
> > acknowledged ("device", or emulated cvq in host vdpa backend driver,
> > has used all cvq buffers, enable (SET_VRING_ENABLE, set_vq_ready) all
> > other queues.
> >
> What is special about doing DRIVER_OK and enqueuing the control
> commands?
> Why other configuration cannot be applied before DRIVER_OK?
> [GD>>] For the device to be live (and any queue to be able to pass traffic),
> DRIVER_OK is a must. 
This applies only to the vdpa device implemented over virtio device.
For such use case/implementation anyway a proper virtio spec extension is 
needed for it be efficient.
And what that happens this scheme will still work.

Other vdpa devices doesn’t have to live with this limitation at this moment.

> So, any configuration can be passed over the CVQ only
> after it is started (vring configuration + DRIVER_OK). For an emulated queue,
> if the order is reversed, I think the enqueued commands will remain
> buffered and device should be able to service them when it goes live.
I likely didn’t understand what you describe above.

Vq avail index etc is programmed before doing DRIVER_OK anyway.

Sequence is very straight forward at destination from user to kernel.
1. Set config space fields (such as virtio_net_config/virtio_blk_config).
2. Set other device attributes (max vqs, current num vqs)
3. Set net specific config (RSS, vlan, mac and more filters)
4. Set VQ fields (enable, msix, addr, avail indx)
5. Set DRIVER_OK, device resumes from where it left off

Steps #1 to #4 can be done multiple times in pre-warm device up case in future.
For now, they can be done once to get things started.