Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface

[Daniel P . Berrangé]

On Tue, Apr 26, 2022 at 01:14:28PM +0200, Markus Armbruster wrote:
> Victor Toso <victortoso@redhat.com> writes:
> 
> > Hi,
> >
> > Happy 1st April. Not a joke :) /* ugh, took me too long to send */
> >
> > This series is about adding a generator in scripts/qapi to produce
> > Go data structures that can be used to communicate with QEMU over
> > QMP.
> >
> >
> > * Why Go?
> >
> > There are quite a few Go projects that interact with QEMU over QMP
> > and they endup using a mix of different libraries with their own
> > code.
> >
> >
> > ** Which projects?
> >
> > The ones I've found so far:
> >
> > - podman machine
> >   https://github.com/containers/podman/tree/main/pkg/machine/qemu
> >
> > - kata-containers (govmm)
> >   
> > https://github.com/kata-containers/kata-containers/tree/main/src/runtime/pkg/govmm
> >
> > - lxd
> >   https://github.com/lxc/lxd/tree/master/lxd/instance/drivers
> >
> > - kubevirt (plain json strings)
> >   https://github.com/kubevirt/kubevirt
> >
> > (let me know if you know others)
> >
> >
> > * But Why?
> >
> > I'm particularly interested in 3 out of 4 of the projects above and
> > only Kubevirt uses libvirt to handle QEMU. That means that every
> > QEMU releases where a QMP command, event or other data struct is
> > added, removed or changed, those projects need to check what changed
> > in QEMU and then address those changes in their projects, if needed.
> >
> > The idea behind generating Go data structures is that we can keep a
> > Go module which can have releases that follow QEMU releases.
> 
> We need to look at "following the QEMU releases" a bit more closely.
> 
> Merging your patches gives us the capability to generate a Go interface
> to HEAD's version of QMP.
> 
> The obvious way for an out-of-tree Go program to use this generated Go
> interface is to build with a specific version of it.  It can then talk
> QMP to any compatible QEMU version.
> 
> Compatibility with older QEMUs is not assured: stuff added since is
> present on the Go QMP client end, but not on the QEMU QMP server end.
> 
> Compatibility with newer QEMUs is subject to our deprecation policy:
> 
>     In general features are intended to be supported indefinitely once
>     introduced into QEMU.  In the event that a feature needs to be
>     removed, it will be listed in this section.  The feature will remain
>     functional for the release in which it was deprecated and one
>     further release.  After these two releases, the feature is liable to
>     be removed.
> 
> So, if you stay away from deprecated stuff, you're good for two more
> releases at least.
> 
> Does this work for the projects you have in mind?

It might work for some projects, but in the general case I find it pretty
unappealing as a restriction. Mixing and matching new QEMU with old libvirt,
or vica-verca has been an incredibly common thing todo when both developing
and perhaps more importantly debugging problems. For example I have one
libvirt build and I use it against any QEMU from Fedora / any RHEL-8.x
update, which spans a great many QEMU releases. 

I like the idea of auto-generating clients from the QAPI schema, and
would like it if we were able to use this kind of approach on the libvirt
side, but for that we need to be more flexible in version matching.

Our current approach to deprecation features and subsequently removing
them from the QAPI schema works fine when the QAPI schema is only used
internally by QEMU, not when we we expand usage of QAPI to external
applications. 

I think we need to figure out a way to make the QAPI schema itself be
append only, while still allowing QEMU to deprecation & remove features.

For a minimum viable use case, this doesn't feel all that difficult, as
conceptually instead of deleting the field from QAPI, we just need to
annotate it to say when it was deleted from the QEMU side.  The QAPI
generator for internal QEMU usage, can omit any fields annotated as
deleted in QAPI schema. The QAPI generator for external app usage,
can (optionally) be told to include deleted fields ranging back to
a given version number. So apps can chooses what degree of compat
they wish to retain.

Apps that wish to have version compat, would of course need to write
their code to be aware of which fields they need to seend for which
QEMU version.


> > * Status
> >
> > There are a few rough edges to work on but this is usable. The major
> > thing I forgot to add is handling Error from Commands. It'll be the
> > first thing I'll work on next week.
> >
> > If you want to start using this Today you can fetch it in at
> >
> >     https://gitlab.com/victortoso/qapi-go/
> >
> > There are quite a few tests that I took from the examples in the
> > qapi schema. Coverage using go's cover tool is giving `28.6% of
> > statements`
> >
> > I've uploaded the a static generated godoc output of the above Go
> > module here:
> >
> >     
> > https://fedorapeople.org/~victortoso/qapi-go/rfc/victortoso.com/qapi-go/pkg/qapi/
> >
> >
> > * License
> >
> > While the generator (golang.py in this series) is GPL v2, the
> 
> I'd make it v2+, just to express my displeasure with the decision to
> make the initial QAPI generator v2 only for no good reason at all.

Our policy is that all new code should be v2+ anyway, unless it was
clearly derived from some pre-existing v2-only code. Upto Victor to
say whether the golang.py is considered clean, or was copy+paste
in any parts from existin v2-only code

> > generated code needs to be compatible with other Golang projects,
> > such as the ones mentioned above. My intention is to keep a Go
> > module with a MIT license.
>
> Meh.  Can't be helped, I guess.

This does make me wonder though whether the license of the QAPI input
files has a bearing on the Go (or other $LANGUAGE) ouput files. eg is
the Go code to be considered a derived work of the QAPI JSON files. I'm
not finding a clearly articulated POV on this question so far.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|