Re: Call for Google Summer of Code 2021 project ideas

[John Snow]

On 1/14/21 7:29 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
>
> > On 1/13/21 4:19 AM, Markus Armbruster wrote:
> > > John Snow <jsnow@redhat.com> writes:
> > >
> > > > On 1/11/21 6:47 AM, Stefan Hajnoczi wrote:
> > > > > Dear QEMU, KVM, and rust-vmm community,
> > > > > QEMU will apply for Google Summer of Code
> > > > > (https://summerofcode.withgoogle.com/) again this year.  This internship
> > > > > program offers paid, 10-week, remote work internships for
> > > > > contributing to open source.  QEMU can act as an umbrella organization
> > > > > for KVM kernel and rust-vmm projects too.
> > > > > Please post project ideas on the QEMU wiki before February 14th:
> > > > > https://wiki.qemu.org/Google_Summer_of_Code_2021
> > > > > What's new this year:
> > > > >     * The number of internship hours has been halved to 175 hours over
> > > > >       10 weeks. Project ideas must be smaller to fit and students will have
> > > > >       more flexibility with their working hours.
> > > > >     * Eligibility has been expanded to include "licensed coding school or
> > > > >       similar type of program".
> > > > > Good project ideas are suitable for 175 hours (10 weeks half-day)
> > > > > work by a
> > > > > competent programmer who is not yet familiar with the codebase.  In
> > > > > addition, they are:
> > > > >     * Well-defined - the scope is clear
> > > > >     * Self-contained - there are few dependencies
> > > > >     * Uncontroversial - they are acceptable to the community
> > > > >     * Incremental - they produce deliverables along the way
> > > > > Feel free to post ideas even if you are unable to mentor the
> > > > > project.
> > > > > It doesn't hurt to share the idea!
> > > >
> > > > I have one that is probably way too ambitious, but requires a
> > > > particular skillset that might be of good interest to a student that
> > > > has some experience in the area already.
> > > >
> > > > The idea is for a TUI qmp-shell (maybe using urwid?) to create an
> > > > irssi-like REPL interface for QMP. The idea would be to mimic the
> > > > mitmproxy TUI interface (Check it out if you haven't!)
> > > >
> > > > All the ideas below are extremely tentative to give a basic gist of
> > > > what I mean; exact layouts/hotkeys/etc are for the sake of explanation
> > > > only.
> > > >
> > > > Essentially, I want an interface like this:
> > > >
> > > > -----------------------------------------------------------
> > > > | QMP Mode                                                |
> > > > |=========================================================|
> > > > |                                                         |
> > > > | Welcome to the Qemu Machine Protocol shell.             |
> > > > | Please type /help or Ctrl+H to see available commands.  |
> > > > |                                                         |
> > > > |                                                         |
> > > > |                                                         |
> > > > |---------------------------------------------------------|
> > > > | >                                                       |
> > > > -----------------------------------------------------------
> > > >
> > > > commands are entered in the bottom and appear in a log window above,
> > > > appearing most-recent last, like irssi works.
> > > >
> > > > As an example, let's say we issue block-dirty-bitmap-add:
> > > >
> > > > --------------------------------------------------------
> > > > | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 |
> > > > --------------------------------------------------------
> > > >
> > > > (...syntax up for debate...! We want something easy to parse, but easy
> > > > to type. For structured data, that's Hard.)
> > > "Hard" is a red flag for such a brief internship.  We need to solve
> > > or
> > > somehow exclude this design problem before we hand the task to the
> > > student.
> > > [...]
> >
> > Yes, there is a reason I capitalized "Hard". :~)
> >
> > This task CAN be split out, though. For starters, we can use an
> > extremely simplified parsing strategy while we work on the REPL and
> > and UI elements.
> >
> > We can accept things of the form:
> > command arg=value arg2=value
> >
> > and we can worry about how to input structured data later.
>
> Bzzzt!  You just designed a language.  The fact that it is obviously
> inadequate for the task does not make this not language design.

Did *I* design a language, or did qmp-shell authors design a language? O:-)

(Oh, I'm one of the authors. Well then.)

> Two outcomes:
>
> 1. The project fails for other reasons.  The language doesn't matter.
>
> 2. The project succeeds, i.e. it becomes useful to people.  Very quickly
> these people will demand you evolve your inadequate language into a less
> inadequate one, compatibly.  Your evasion of the language design problem
> (because hard) has now added accidental constraints, making the design
> problem even harder.

There's no reason that an interactive shell has to stay backwards 
compatible. Version 0.1 and 0.2 might offer different input paradigms. 
It's not nice, but it's fine for beta-level stuff.

Still, I appreciate that "Beta! Might change!" is an oft-ignored 
warning. This isn't a library or API though, so I still think it's okay 
to take some liberty here.

> The sane way to evade the language design problem is to use the existing
> QMP language.

I wouldn't mind implementing this for version 0.1 -- just allow 
copy-pasting JSON into the input bar -- it's a feature I wanted anyway.

Some things I would like to see, but suspect are not reasonably possible 
to implement quickly:

1. A shorthand syntax mode similar to qmp-shell that is reasonably 
sufficient for most commands. This involves solving a fairly difficult 
language design problem on how to facilitate the input of structured 
data. The biggest pain point here is nested data; we allow JSON as an 
input here but require that it has no spaces. We do not offer a "native" 
shorthand for the nested data.

We *could* co-opt the dotted key syntax, but that's cumbersome for some 
constructs because it involves a lot of repeated parent keys.

As you say, the language design aspect is hard.


2. a multi-line editor mode that facilitates typing QMP messages that 
offers syntax highlighting, field and type validation, autocomplete, 
etc. I have no idea how to implement this -- it's basically an entire 
bespoke editor.


That's *very* hard stuff, I expect, so I won't go there for starters.

So I see two possible options for "not inventing a language":

1. Raw QMP
2. The existing qmp-shell syntax, warts and all.

I don't see a tremendous problem with doing both; but we can start with 
raw QMP. The existing qmp-shell syntax is at least definitely very easy 
to write a new parser for, even if it's kind of ugly and insufficient. I 
still see value in designing a new TUI with the old syntax.

> The project then aims to build a tool that adds useful features over
> "socat "READLINE,history=$HOME/.qmp_history,prompt=QMP>"
> UNIX-CONNECT:/path/to/socket".
>
> If it succeeds, you can still design and implement a "better" language,
> and let users choose the one they prefer.  Or you could add features to
> help with typing QMP.
>
> >                                                             I don't
> > think it's a blocker to have someone work on the TUI and asynchronous
> > dispatch elements. I think even just keeping our current parsing but
> > adding some of the features outlined in the proposal would be a big
> > usability win.
>
> I don't feel this particular itch, but I'm certainly not objecting to
> anyone scratching.

It's something I'd like to see so that I can walk non-QEMU devs through 
interacting with QEMU at a low level for the purposes of debugging, 
reproducing problems, prototyping features, etc.

I use qmp-shell all the time for debugging things myself, I find it 
easier to use than copy-pasting things directly into socat. I wouldn't 
mind the shell getting a little smarter to help me out -- the ability to 
see async events and reconnect on disconnect would already be a massive 
improvement to *my* quality of life.

So much so that I spent a lot of time in December to write an async qmp 
library O:-)

--js