Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Akira Kyle]

On Sat, Nov 28, 2020 at 01:38 AM, Eli Zaretskii <eliz@gnu.org> 
wrote:

> > > It is a relatively simple matter to disable some 
> > > optimizations
> > > when a widget is displayed.  It is also relatively simple to 
> > > tell
> > > the widget to redraw itself when redisplay changed its 
> > > position on
> > > the screen.
> >
> > Yes, but you had previously pointed to that fact that xwidgets 
> > needs to such disable such optimizations as part of the 
> > "kludge" 
> > that would need to be resolved for xwidgets to be considered 
> > viable for a path forward. Having looked at that, I'm not sure 
> > gtk 
> > will ever want to cooperate without doing things that would be 
> > considered hacks by gtk folks.
>
> I'd appreciate more details on the GTK part of this.  I only 
> said that
> from the Emacs display engine side things are not too hairy. 
> from my
> POV, xwidgets integration simply stopped short of going all the 
> way
> towards integration into the redisplay framework, and I don't 
> think
> anyone said before that this was done because better integration 
> was
> impossible or very hard.

One such fundamental incompatibility stems from the fact that 
Emacs may have multiple views on one buffer split across various 
windows and/or frames. Gtk assumes that each widget will only have 
one view. The 'webkit xwidget type currently works around this by 
drawing to an offscreen window then for every window that's 
supposed to display the widget, creating a gtk drawing area widget 
and copying the offscreen window's surface to the drawing 
surface. I don't think gtk really intends for ofscreen widgets to 
be used this way and as far as I can tell offscreen window widgets 
have been removed in gtk4.

Another issue is that emacs uses a GtkFixed (or a custom 
subclassed EmacsFixed for gtk3) as the container widget for the 
main frame area. There isn't a mechanism to control z-ordering of 
child widgets other than removing and re-adding them in the 
desired z-order, which is what emacs-webkit has to do in order for 
child frames to not render below the webkit widget.

> > > I don't see why you consider this such a significant problem. 
> > > The
> > > number of interfaces that such a "rich" display element needs 
> > > to
> > > support is very small and well-defined.  The problem with 
> > > xwidgets
> > > in this regard is that its interaction with the display code 
> > > was
> > > left unfinished, it basically just copy/pastes the code which
> > > supports display of images, which is not 100% correct, since
> > > xwidgets don't display a static bitmap.
> >
> > The "number interfaces" will likely be small no matter what 
> > solution is taken, its more about how easy you want to make it 
> > for 
> > a "rich" display element to really mess with Emacs' display and 
> > input handling. For example with xwidgets the onus is on the 
> > widget implementation to not totally steal keyboard focus from 
> > Emacs. I've found that gtk widgets really like stealing 
> > keyboard 
> > focus so its easy to end up in a state you can't escape out 
> > of.
>
> Keyboard focus is a separate issue.  I only talked about 
> integration
> with redisplay.
>
> For keyboard input integration, we'd need to develop some
> infrastructure, if the existing one doesn't provide enough 
> hooks.  If
> you or someone else can describe the details, we could discuss 
> that in
> more practical terms.  I think it's too soon to give up on the
> practical possibility to solve these problems before we 
> discussed
> that, or are even fully aware of the problems and the potential
> solutions.
>
> > Similarly gtk widgets won't always easily draw themselves 
> > where you want at the size you want so its easy for widgets to 
> > render, say, across window boundaries.
>
> If that is true, then your method of letting the widget display 
> in a
> window won't work well either, right?  IOW, whether the screen 
> area
> given to the widget is a full Emacs window or just its part is 
> not an
> important aspect of the integration of such widgets into the 
> Emacs
> display.
>
> > The xwidget approach requires non-insignificant work on the 
> > widget
> > end to ensure the widgets will work within the confines of 
> > Emacs'
> > redisplay.
>
> As long as the widget can be told to use a given rectangular 
> portion
> of the screen, we should be fine: the current Emacs display 
> engine is
> perfectly capable of handling display elements that occupy an 
> area of
> certain dimensions.

This is more an issue on the gtk side of things. With the webkit 
widget it isn't an issue since the widget doesn't define a minimum 
size so any size you request it to render at, it will 
obey. However other widgets, such as buttons, sliders, etc, have a 
minimum size they will render at. If you tell it to render itself 
smaller, it will refuse and so it's impossible to simply tell the 
widget "here's the space you have to go in". This becomes a 
problem when the widget needs to be clipped at a window boundary 
as gtk will happily draw the widgets across them as gtk has no 
internal knowledge of Emacs window boundaries being a place that 
widget must be clipped at. Thus every widget that will live in a 
buffer will need to be inside a custom container than handles this 
clipping for the widget.

> > I'd think that such an interface to support "rich" display 
> > elements
> > should be restrictive enough to not easily allow the "rich" 
> > display
> > elements to misbehave.
>
> Please tell more about the reasons for this conclusion.  I don't 
> yet
> see the difficulties you envision.

While all these issues can be overcome in one way or another, I 
think they point at the differences both the UI paradigms of Emacs 
and gtk and the subsequent differences in how each want to draw to 
the screen. In addition to the above specific examples of 
difficulties I've encountered so far with making gtk widgets work 
well with Emacs' display, there's the fundamental problem of this 
route for "richer display elements" not being portable. It'll 
always be confined to Emacs builds `--with-x` (or `--with-pgtk` 
now). While I feel this is fine for emacs-webkit, I think it would 
be advantageous to not tie such a feature to gtk itself.

The direction I'm thinking of going in the future since I feel 
like thus far the xwidget route has been fairly troublesome is to 
expose interfaces to the cairo surfaces used in image.c. This 
could be through modifying image.c or through defining a new 
display element type. I would try to expose the cairo API to lisp 
and allow dynamic modules to directly draw to a cairo surface 
given to it through the lisp interface. Gtk along with many other 
libraries (such as librsvg and poppler) support drawing to a cairo 
surface and then Emacs gets to have full control over how that 
cario surface ends up on screen. I would see this as a stepping 
stone to attempting to do something similar with a opengl surface.