Re: [RFC PATCH v1 0/8] qapi: add generator for Golang interface

[Markus Armbruster]

Victor Toso <victortoso@redhat.com> writes:

> Hi,
>
> On Wed, May 11, 2022 at 04:17:35PM +0200, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> > Caller
>> >
>> >     block_resize(device="dev0", size=1*GiB)
>> >     block_resize(node_name="devnode0", size=1*GiB)
>> >
>> >
>> > In golang definition
>> >
>> >    type BlockResizeArguments struct {
>> >        Device string
>> >        NodeName string
>> >        Size int
>> >    }
>> >
>> > Caller choice of
>> >
>> >    cmd := &BlockResizeCommand{
>> >        Device: "dev0",
>> >        Size: 1 * GiB,
>> >    }
>> >
>> >    cmd := &BlockResizeCommand{
>> >        NodeName: "devnode0",
>> >        Size: 1 * GiB,
>> >    }
>> 
>> Note that the Go bindings you sketched effectively use (poor
>> man's) keyword arguments.
>> 
>> > Neither case can easily prevent passing Device and NodeName
>> > at same time.
>> 
>> That defect lies at the schema's feet.
>
> Right. The schema does not provide any metadata to explicit say
> that only @device or @node-name should be used, correct?

Correct.

The existing means to express either / or are tagged unions and
alternates.

Tagged unions require an explicit tag member.

Alternates select the alternative by the type of the value.

We don't have anything that selects by member name.

> This would be important to differentiate of a simple 'adding a
> new optional argument' plus 'making this other argument
> optional'.

We also don't have means to express "this integer must be a power of
two", or "this string must name a block backend", or any number of
semantic constraints.

We have to draw the line somewhere.  Schema language complexity needs to
earn its keep.

>> >> * At some future date, the old way gets deprecated: argument @device
>> >>   acquires feature @deprecated.
>> >
>> > Ok, no change needed to the APIs in either case. Possibly have
>> > code emit a warning if a deprecated field is set.
>> >
>> >> * Still later, the old way gets removed: @device is deleted, and
>> >>   @node-name becomes mandatory.
>> >
>> > Again no change needed to APIs, but QEMU will throw back an
>> > error if the wrong one is used. 
>> >
>> >> What is the proper version-spanning interface?
>> >> 
>> >> I figure it's both arguments optional, must specify the right one for
>> >> the version of QEMU actually in use.  This spans versions, but it fails
>> >> to abstract from them.
>> >
>> > Yep, I think that's inevitable in this scenario. THe plus side
>> > is that apps that want to span versions can do so. The downside
>> > is that apps that don't want smarts to span version, may loose
>> > compile time warnings about use of the now deleted field. 
>> 
>> The version-spanning interface will arguably be a bad interface for any
>> version.
>>
>> > I suggested the code generator have an option to say what level
>> > of compat to use for generated code, so that apps can request an
>> > API without compat, which will result in compile errors. This
>> > though assumes every consumer app is embedding their own
>> > generated copy of the code. Not neccessarily desirable.
>> >
>> > At the C level you can play games with __deprecated__ to get
>> > compile time warnings in some cases. 
>> >
>> > #define GLIB_VERSION_MIN_REQUIRED GLIB_VERSION_2_56
>> >
>> > causes QEMU to get compile time warnings (or errors) if it
>> > attempts to use a API feature deprecated in 2.56, even if
>> > the API exists in the header & library. 
>> >
>> >
>> >> Note that it's not enough to replace "delete member" by "mark member
>> >> deleted in <version>".  You also have to keep full history for "is it
>> >> optional".  And for types, because those can evolve compatibly, too,
>> >> e.g. from struct to flat union, or from string to alternate of string
>> >> and something else.  What is the proper version-spanning interface in
>> >> all the possible cases?
>> >
>> > I've not thought through all possible scenarios, but there may end
>> > up being restrictions, such that changes that were previously possible
>> > may have to be forbidden.
>> 
>> "There may be restrictions" is not exactly a confidence-inspring design
>> assumption.  We need a reasonably dependable idea on what exactly we're
>> intending to sacrifice.
>
> I can't help much here but I guess we can evolve QAPI schema as
> we move forward. Adding metadata that helps document changes to
> the benefit of giving code generators tools to provide a way to
> work with those QAPI changes seems desirable, no?

QMP comes with a certain compatibility promise.  Keeping the promise has
been expensive, but that's okay; a weaker promise would have been
differently expensive.  It's a compromise that has emerged over a long
time.

I fail to see why QEMU-provided QMP bindings should come with a stronger
promise than QEMU-provided QMP.

>> > One example,  in the past we could do deprecate a field 'foo', then
>> > delete 'foo' and then some time re-introduce 'foo' with a completely
>> > different type. That would not be possible if we wanted to maintain
>> > compat, but in this example that's probably a good thing, as it'll
>> > be super confusing to have the same field name change type like that
>> > over time. Easier to just use a different name.
>> >
>> > So the question to me is not whether all our previous changes are
>> > still possible, but whether enough of the typwes of change are
>> > possible, such that we can cope with the ongoing maint in a
>> > reasonable way. I don't think we've explored the possibility enough
>> > to say one way or the other.
>> >
>> >> > Apps that wish to have version compat, would of course need to write
>> >> > their code to be aware of which fields they need to seend for which
>> >> > QEMU version.
>> >> 
>> >> At which point we're reinventing libvirt.
>
> IMHO, at this moment, qapi-go is targeting communicating with
> QEMU and handling multiple QEMU versions seems reasonable to me.

It's targeting communicating in *QMP*.  QMP is designed to support
communicating with a range of QEMU versions.  Full compatibility is
promised for a narrow range.  Outside that range, graceful degradation.

*If* you want to widen the full compatibility range, do it in *QMP*.  Or
do it on top of QEMU, e.g. in libvirt.

> Perhaps libvirt can use qapi-go in the future or other generated
> interface. That would be cool.

"Would be cool" and a dollar buys you a cup of bad coffee.

Is it a good use of our limited resources?

How much will it delay delivery of Go bindings compared to less
ambitious version?

>> > The premise of the code generators is that there are applications
>> > that want to consume QEMU that are not libvirt. With this line of
>> > reasoning we could easily say that all such applications should
>> > just use libvirt and then we don't need to provide any of these
>> > code generators.  The fact that we're considering these code
>> > generators though, says that we're accepting there are valid use
>> > cases that don't want to use libvirt for whatever reasons.
>> 
>> Can't resist any longer: "What has libvirt ever done for us?"
>> https://www.youtube.com/watch?v=Qc7HmhrgTuQ
>> 
>> >                                                            It is
>> > reasonable that some of those applications may wish to target
>> > a wide range of QEMU versions, just like libvirt does.
>> 
>> At which point they're comitting to reinventing the relevant parts of
>> libvirt.
>> 
>> I'd expect marshalling and umarshalling QMP to be among the
>> smaller sub-problems then.  It may look relatively large at
>> first, because it's among the first ones to be solved.  More so
>> when you hand-wave away the more interesting ones of
>> *abstraction* until they bite you in the posterior.
>
> I might have missed it but I don't see unsolvable problems.
>
>  1) We decide if we want a Golang interface that can communicate
>     with multiple QEMU versions or not;
>  2) We discuss how that Golang interface would look like;
>  3) We discuss what is needed in QEMU/QAPI to achieve (2)
>  4) We work on QEMU/QAPI to address (3)
>  5) We move on with qapi-go proposal

This reinvents significant parts of libvirt in QEMU.

> I see only benefits with this project, plus the fact we already
> have Golang projects doing their own code to communicate with
> QEMU and I do believe we will make their lives easier.

Perhaps my idea of practical Go usage is hopelessly naive, but here goes
anyway.

Go project vendors current Go bindings (because they vendor everything).
Works with current QEMU, plus a range of past and future QEMUs.

Eventually something breaks with a then current QEMU.  Go project
updates vendored Go bindings to current.  Compiler points out everything
they need fixing (this assumes the type system is sufficiently
expressive, and bindings use it properly).

Good enough for these Golang projects?

Or asked in a different way: what exactly are the compatibility
requirements of these projects?

Please keep it to requirements *actual* projects have.  Providing for
additional, non-trivial requirements hypothetical projects might have
is, in my reasoned opinion, folly we cannot afford.  YAGNI until
demonstrated otherwise.

If different projects have different requirements, then QEMU having to
provide for all of them does not follow.  Nothing absolves us from
weighing benefits vs. costs and risks.