Re: [PATCH v2 2/7] multifd: modifying 'migrate' qmp command to add multifd socket on particular src and dest pair

[Daniel P . Berrangé]

On Thu, Jul 28, 2022 at 08:32:39PM +0530, Het Gala wrote:
> 
> On 26/07/22 4:43 pm, Daniel P. Berrangé wrote:
> > On Thu, Jul 21, 2022 at 07:56:15PM +0000, Het Gala wrote:
> > > i) Modified the format of the qemu monitor command : 'migrate' by adding 
> > > a list,
> > >     each element in the list consisting of multifd connection parameters: 
> > > source
> > >     uri, destination uri and of the number of multifd channels between 
> > > each pair.
> > > 
> > > ii) Information of all multifd connection parameters' list and length of 
> > > the
> > >      list is stored in 'OutgoingMigrateParams' struct.
> > > 
> > > Suggested-by: Manish Mishra <manish.mishra@nutanix.com>
> > > Signed-off-by: Het Gala <het.gala@nutanix.com>
> > > ---
> > >   migration/migration.c | 52 +++++++++++++++++++++++++++++--------
> > >   migration/socket.c    | 60 ++++++++++++++++++++++++++++++++++++++++---
> > >   migration/socket.h    | 19 +++++++++++++-
> > >   monitor/hmp-cmds.c    |  1 +
> > >   qapi/migration.json   | 47 +++++++++++++++++++++++++++++----
> > >   5 files changed, 160 insertions(+), 19 deletions(-)
> > > 
> > > diff --git a/qapi/migration.json b/qapi/migration.json
> > > index 81185d4311..456247af8f 100644
> > > --- a/qapi/migration.json
> > > +++ b/qapi/migration.json
> > > @@ -1449,12 +1449,37 @@
> > >   ##
> > >   { 'command': 'migrate-continue', 'data': {'state': 'MigrationStatus'} }
> > > +##
> > > +# @MigrateUriParameter:
> > > +#
> > > +# Information regarding which source interface is connected to which
> > > +# destination interface and number of multifd channels over each 
> > > interface.
> > > +#
> > > +# @source-uri: uri of the source VM. Default port number is 0.
> > > +#
> > > +# @destination-uri: uri of the destination VM
> > > +#
> > > +# @multifd-channels: number of parallel multifd channels used to migrate 
> > > data
> > > +#                    for specific source-uri and destination-uri. 
> > > Default value
> > > +#                    in this case is 2 (Since 7.1)
> > > +#
> > > +##
> > > +{ 'struct' : 'MigrateUriParameter',
> > > +  'data' : { 'source-uri' : 'str',
> > > +             'destination-uri' : 'str',
> > > +             '*multifd-channels' : 'uint8'} }
> > > +
> > >   ##
> > >   # @migrate:
> > >   #
> > >   # Migrates the current running guest to another Virtual Machine.
> > >   #
> > >   # @uri: the Uniform Resource Identifier of the destination VM
> > > +#       for migration thread
> > > +#
> > > +# @multi-fd-uri-list: list of pair of source and destination VM Uniform
> > > +#                     Resource Identifiers with number of 
> > > multifd-channels
> > > +#                     for each pair
> > >   #
> > >   # @blk: do block migration (full disk copy)
> > >   #
> > > @@ -1474,20 +1499,32 @@
> > >   # 1. The 'query-migrate' command should be used to check migration's 
> > > progress
> > >   #    and final result (this information is provided by the 'status' 
> > > member)
> > >   #
> > > -# 2. All boolean arguments default to false
> > > +# 2. The uri argument should have the Uniform Resource Identifier of 
> > > default
> > > +#    destination VM. This connection will be bound to default network
> > >   #
> > > -# 3. The user Monitor's "detach" argument is invalid in QMP and should 
> > > not
> > > +# 3. All boolean arguments default to false
> > > +#
> > > +# 4. The user Monitor's "detach" argument is invalid in QMP and should 
> > > not
> > >   #    be used
> > >   #
> > >   # Example:
> > >   #
> > > -# -> { "execute": "migrate", "arguments": { "uri": "tcp:0:4446" } }
> > > +# -> { "execute": "migrate",
> > > +#      "arguments": {
> > > +#          "uri": "tcp:0:4446",
> > > +#          "multi-fd-uri-list": [ { "source-uri": "tcp::6900",
> > > +#                                   "destination-uri": "tcp:0:4480",
> > > +#                                   "multifd-channels": 4},
> > > +#                                 { "source-uri": "tcp:10.0.0.0: ",
> > > +#                                   "destination-uri": 
> > > "tcp:11.0.0.0:7789",
> > > +#                                   "multifd-channels": 5} ] } }
> > >   # <- { "return": {} }
> > >   #
> > >   ##
> > >   { 'command': 'migrate',
> > > -  'data': {'uri': 'str', '*blk': 'bool', '*inc': 'bool',
> > > -           '*detach': 'bool', '*resume': 'bool' } }
> > > +  'data': {'uri': 'str', '*multi-fd-uri-list': ['MigrateUriParameter'],
> > > +           '*blk': 'bool', '*inc': 'bool', '*detach': 'bool',
> > > +           '*resume': 'bool' } }
> > Considering the existing migrate API from a QAPI design POV, I
> > think there are several significant flaws with it
> > 
> > The use of URIs is the big red flag. It is basically a data encoding
> > scheme within a data encoding scheme.  QEMU code should be able to
> > directly work with the results from QAPI, without having todo a
> > second level of parsing.
> > 
> > URIs made sense in the context of HMP or the QemuOpts CLI, but do not
> > make sense in QMP. We made a mistake in this respect when we first
> > introduced QMP and implemented 'migrate'.
> > 
> > If we going to extend the migrate API I think we should stop using URIs
> > for the new fields, and instead define a QAPI discriminated union for
> > the different data transport backends we offer.
> > 
> >       { 'enum': 'MigrateTransport',
> >         'data': ['socket', 'exec'] }
> > 
> >       { 'union': 'MigrateAddress',
> >         'base': { 'transport': 'MigrateTransport'},
> >         'discriminator': 'transport',
> >         'data': {
> >             'socket': 'SocketAddress',
> >        'exec': ['str'],
> >         }
> > 
> > NB, 'socket' should be able to cover all of  'tcp', 'unix', 'vsock'
> > and 'fd' already. I'm fuzzy on best way to represent RDMA.
> > 
> > 
> > IIUC, the desire of migration maintainers is that we can ultimately
> > have multifd as the preferred, or even only, mechanism. Aside from
> > the main outbound migration control channel, and the multifd
> > data channels, IIUC we have a potential desire to have more channels
> > for post-copy async requests.
> > 
> > This all suggests to me a more general representation along the
> > lines of:
> > 
> >    { 'enum': 'MigrateChannelType',
> >      'data': ['control', 'data', 'async'] }
> > 
> >    { 'struct': 'MigrateChannel',
> >      'data': {
> >         'type': 'MigrateChannelType',
> >         'src-addr': 'MigrateAddress',
> >         'dst-addr': 'MigrateAddress',
> >         'count': 'int',
> >      } }
> > 
> >    { 'comand': 'migrate',
> >      'data': {
> >        '*uri': 'str'
> >        '*channels': ['MigrateChannel']
> >      }
> >    }
> > 
> > With 'uri' and 'channels' being mutually exclusive here.
> > 
> > This whole scheme brings in redundancy wrt to the 'migrate-set-parameters'
> > API wrt multifd - essentally the same data is now being set in two
> > different places. IMHO, we should declare the 'multifd' capability
> > and the 'multifd-chanels' parameter deprecated, since the information
> > they provide is totally redundant, if you're giving an explicit list
> > of channels to 'migrate'.
> > Hi Daniel. Initially while brainstorming this idea for the first time, we
> also came up with the same thought of depricating the migrate API. But how
> will we achieve this now and how is it going to work. Is it like we will be
> making migate V2 APIs initially, integrate it and then depricate the old
> one? would be happy to get some pointers from your end.

I don't think we need to replace 'migrate' with 'migrate2', because our
QAPI deprecation & feature addition policy lets us evolve the existing
commands over time.

We currently have

    { 'comand': 'migrate',
      'data': {
        'uri': 'str'
      }
    }

Our ABI policy lets us make 'uri' optional, and as well as adding new
optional fields:

     { 'comand': 'migrate',
       'data': {
         '*uri': 'str'
         '*channels': ['MigrateChannel']
       }
     }

At some point we can choose to deprecate 'uri' entirely and just ask
people to use a single element 'channels'  array. After having 'uri'
deprecated for a while we can delete it and make 'channels' mandatory

     { 'comand': 'migrate',
       'data': {
         'channels': ['MigrateChannel']
       }
     }


This gives us the same end point as if we had added

     { 'comand': 'migrate2',
       'data': {
         'channels': ['MigrateChannel']
       }
     }

and deprecated &deleted 'migrate' entirely, but without leaving us with
the ugly naming.


FWIW, if I was going to majorly refactor migration QAPI design, I would
also seek to get rid of 'migrate-set-capability' entirely and just
make it a parameter to 'migrate', and likewise for the same for any of
'migrate-set-parameter' that are only used as upfront tunables. We
should only need 'migrate-set-parameter' for runtime tunables that need
setting /after/ migration has already started. I didn't mention this
before though, as I didn't want to impose a big chunk of extra work
on this feature of yours, as it can be done separately at any time.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|