Re: Call for Google Summer of Code 2021 project ideas

[John Snow]

On 1/11/21 6:47 AM, Stefan Hajnoczi wrote:
> Dear QEMU, KVM, and rust-vmm community,
> QEMU will apply for Google Summer of Code
> (https://summerofcode.withgoogle.com/) again this year.  This internship
> program offers paid, 10-week, remote work internships for
> contributing to open source.  QEMU can act as an umbrella organization
> for KVM kernel and rust-vmm projects too.
>
> Please post project ideas on the QEMU wiki before February 14th:
> https://wiki.qemu.org/Google_Summer_of_Code_2021
>
> What's new this year:
>   * The number of internship hours has been halved to 175 hours over
>     10 weeks. Project ideas must be smaller to fit and students will have
>     more flexibility with their working hours.
>   * Eligibility has been expanded to include "licensed coding school or
>     similar type of program".
>
> Good project ideas are suitable for 175 hours (10 weeks half-day) work by a
> competent programmer who is not yet familiar with the codebase.  In
> addition, they are:
>   * Well-defined - the scope is clear
>   * Self-contained - there are few dependencies
>   * Uncontroversial - they are acceptable to the community
>   * Incremental - they produce deliverables along the way
>
> Feel free to post ideas even if you are unable to mentor the project.
> It doesn't hurt to share the idea!

I have one that is probably way too ambitious, but requires a particular 
skillset that might be of good interest to a student that has some 
experience in the area already.

The idea is for a TUI qmp-shell (maybe using urwid?) to create an 
irssi-like REPL interface for QMP. The idea would be to mimic the 
mitmproxy TUI interface (Check it out if you haven't!)

All the ideas below are extremely tentative to give a basic gist of what 
I mean; exact layouts/hotkeys/etc are for the sake of explanation only.

Essentially, I want an interface like this:

-----------------------------------------------------------
| QMP Mode                                                |
|=========================================================|
|                                                         |
| Welcome to the Qemu Machine Protocol shell.             |
| Please type /help or Ctrl+H to see available commands.  |
|                                                         |
|                                                         |
|                                                         |
|---------------------------------------------------------|
| >                                                       |
-----------------------------------------------------------

commands are entered in the bottom and appear in a log window above, 
appearing most-recent last, like irssi works.

As an example, let's say we issue block-dirty-bitmap-add:

--------------------------------------------------------
| > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 |
--------------------------------------------------------

(...syntax up for debate...! We want something easy to parse, but easy 
to type. For structured data, that's Hard.)

we press [Enter] to submit the command, and the history now shows:

-------------------------------------------------
| QMP Mode                                      |
|===============================================|
|                                               |
|                                               |
|                                               |
| [11:28] block-dirty-bitmap-add [Pending ...]  |
|-----------------------------------------------|
| >                                             |
-------------------------------------------------

After a few moments, the command resolves:

----------------------------------------
| QMP Mode                             |
|======================================|
|                                      |
|                                      |
|                                      |
| [11:28] block-dirty-bitmap-add [OK]  |
----------------------------------------
| >                                    |
----------------------------------------

The basic unit of history here is either an RPC call/response pair or an 
asynchronous QMP event.

Clicking on the history pane or pressing Alt+↑ or Alt+↓ (Mimics irssi 
keys for switching buffers, let's say it's user-configurable) to change 
focus to the history pane:

------------------------------------------------------------
| Command History Mode                                     |
|==========================================================|
|                                                          |
|                                                          |
|                                                          |
| [11:28] block-dirty-bitmap-add [OK]                      |
-----------------------------------------------------------|
| Use ↑ or ↓ to select a command, press [Enter] to inspect |
------------------------------------------------------------

Using the arrow keys, you can highlight the history item and then press 
<Enter> to change the log pane to a details pane for that item, showing:

------------------------------------------------------
| Details: block-dirty-bitmap-add                [X] |
|====================================================|
|                                                    |
| [11:28:01] -> {                                    |
|    "execute": "block-dirty-bitmap-add",            |
|    "arguments": {                                  |
|      "node": "ide0hd0",                            |
|      "name": "mybitmap0"                           |
|    }                                               |
| }                                                  |
|                                                    |
| [11:28:02] <- {                                    |
|   "return": {}                                     |
| }                                                  |
|                                                    |
|----------------------------------------------------|
| Press 'Q' or Backspace to close details pane       |
------------------------------------------------------

From here, maybe you could do a few things, like press a button to view 
the raw QMP I/O for the purposes of copy-pasting elsewhere, or just 
press Q/H/Backspace to go back to the history view.


Interesting abilities this style of shell can provide:

(1) The ability to render asynchronous events from the server in the log 
window without overwriting the user's command buffer. (In contrast to 
qmp-shell today, which cannot show async events until the user presses 
enter again to prompt a check for them.)

(2) The ability to asynchronously reconnect to servers on connection 
failures.

(3) The ability to collapse command:response pairs into one summarized 
line in the log, which can be expanded for more information later with a 
keypress.

(4) (As a future add-on, beyond current scope) The ability to add a 
status pane widget that might show current running jobs, the current 
machine execution status, etc.



As options, users should be able to:

1. Choose the history display mode live at runtime:
  a) Summarized command history (as shown above)
  b) Summarized JSON history
     (one item per message, no grouping, purely linear,
      shows a one-line summary of the message.)
  c) Full JSON history
     (one item per message, JSON messages are rendered in full.)
  d) Pretty JSON history
     (one item per message, JSON messages are pretty-printed.)

  e.g. Ctrl+V could change the "verbosity" of the messages, and the
  history log will be re-rendered to reflect the new setting.

2. Customize hotkeys for switching between command mode and history 
mode; quitting the details view, etc.

3. toggle on/off automatic command execution logging; or manually 
(Ctrl+S?) save history to file. The format might need some light markup 
to indicate directionality, timestamps, and so on.



Other things that might be of interest, but are beyond the scope of a 
GSoC/Outreachy internship:

1. readline-esque hotkey functionality for the command input bar, 
including command history, autocomplete for command names, etc. If we 
aren't using readline itself anymore, these things will have to be 
implemented somehow. There are libraries that support this: 
https://github.com/rr-/urwid_readline

2. Recording execution macros; subsets of commands you wish to replay 
easily.

3. Raw input mode: allow the user to paste a raw JSON command directly 
into the command bar.

4. Configuring our GTK interface to open up qmp-shell inside one of the 
terminal panes. (Kevin Wolf suggested this or something like it once, 
and I rather like the idea.)

5. "Shell Builtins" -- analogous to the same in bash -- additional 
commands and routines that help perform certain actions that might 
require complex, multi-part steps.

6. Augmented commands -- commands that don't require multi-part phases, 
but ones in which the shell might offer enhanced readouts for. An 
example would be integrating Vladimir's block graph rendering script to 
automatically fire off when block-query is issued. (If the user has a 
graphical session running, the libraries are installed, etc.)



See alsos; prior art, &c:

- irssi as an example of continuous, asynchronous history:

https://irssi.org/

- mitmproxy, which heavily inspired this idea with its use of urwid to 
show a history of "packets" which can be interrogated later from within 
the interface

https://mitmproxy.org/

- aioconsole -- an async python REPL loop for interactively writing 
async code in python. It might have good ideas to steal!

https://pypi.org/project/aioconsole/

- urwid: a TUI framework for Python; it is used by mitmproxy to 
implement its interface.

http://urwid.org/

- urwid readline library: implements readline-esque hotkeys for urwid, 
but (maybe) doesn't respect readline config, if any. It is a 
reimplementation.

https://github.com/rr-/urwid_readline





Target skillset for students:
- Intermediate Python
- Some exposure to async programming, coroutines, etc.
- Some experience with GUI programming concepts, in any toolkit or language
- A willingness to use gradually typed python O:-)

I don't expect this project to involve much, if any, C; but familiarity 
with the linux shell ecosystem will be an asset when choosing default 
behaviors, mimicking other tools, etc.

I have already written a fully asyncio QMP library; I would expect this 
new library to be used. It should be fully operational, but might 
require API changes when we begin to use it for a serious endeavor.

> I will review project ideas and keep you up-to-date on QEMU's
> acceptance into GSoC.
>
> For more background on QEMU internships, check out this video:
> https://www.youtube.com/watch?v=xNVCX7YMUL8
>
> Stefan