Re: Introducing emacs-webkit and more thoughts on Emacs rendering (was Rethinking the design of xwidgets)

[Eli Zaretskii]

> From: Tomas Hlavaty <tom@logand.com>
> Cc: emacs-devel@gnu.org
> Date: Wed, 02 Dec 2020 21:37:08 +0100

It turns out we've been mis-communicating all the time, since I didn't
realize you were interested only in displaying images on TTY frames.
So please forget (almost) everything I said, because most of it is not
relevant.  Below I answer only the questions that are relevant to your
goal.

> At the moment, in emacs-framebuffer, I just call it to display an image
> using call-process-region.  It could be optimized with pipe, but this
> has low priority for emacs-framebuffer as it is fast enough for my
> needs.

IMNSHO, neither call-process-region nor pipes are fast enough to be an
integral part of the Emacs redisplay: they are too slow.  Especially
if you want to be able to display more than one image at the same
time.  To see how the performance is inadequate, turn on flyspell-mode
and scroll through a large text buffer, preferably with lots of
mis-spelled words (or some text full of special terminology the
general-purpose dictionary won't know about).  The effect in your case
will be very similar.

> I am not exatly sure, what the concern is here.
> 
> The images do not travel via pipe.  Only meta-data like filenames,
> coordinates and handles.

Yes, I understood that much.  But that doesn't matter: communicating
with a spell-checker in the flyspell-mode case also sends just a small
number of words in each direction, and yet the slow-down is
considerable, sometimes downright annoying.

> What exactly needs to be done in the middle of redisplay that is
> relevant to drawing images?  Determining image width and height?

Sending the image spec and receiving the dimensions and other image
attributes, yes.  And I presume that the sub-process needs some time
to calculate the answer (because at least some queries require to load
and process an image).

> > You are supposed to submit data in one of the known formats.  Why is
> > that a problem?
> 
> Because they do not work on the console.
> 
> And the reason they do not work on the console is because there is
> hardcoded dependency on image drawing libraries which do not work on the
> console.

No, that's not the main reason.  See below.

> > On the console you currently cannot have images in Emacs, period.
> 
> Exactly.  And I want this to change.  There is no reason for such
> restriction.
> 
> > This is not because of image libraries, this is because the text-mode
> > display simply cannot currently support such display elements.
> 
> This is not an issue at all.  Capabilities of text-mode display are
> irrelevant.  w3mimgdisplay is using framebuffer.  I do not know how to
> better communicate that.

I was talking about the Emacs text-mode display code, sorry for being
unclear.  The Emacs display code which handles the console is not
capable of supporting images, or any display element whose dimensions
are not the same as character cells on a console.  The assumption that
each display element takes exactly one character cell on display is
implicitly present in many places in the Emacs display code that
handles TTYs.  We cannot even display a larger font on a console.
(There is a special handling of wide characters that take two columns
on display -- but still the same height as other characters -- but
that's all.)

To be able to display images on a console within an Emacs text-mode
frame, you need to be able to tell the Emacs display code that a
certain area on display is "taken".  But since images will generally
be larger than a character, in both dimensions, you need the Emacs
text-mode display code to support display elements of arbitrary
dimensions, and that is currently not possible.  Adding that would
need serious changes in the text-mode parts of the Emacs display
engine code, and that is almost entirely in C.

This is the main obstacle we'd need to overcome if Emacs will ever be
able to display images on TTYs.  All the rest is much easier.

> Try:
> 
>    $ w3m https://gnu.org
> 
> You will see text-mode HTML browser layout with images on the console.

I know that text-mode display devices can display graphics, I had it
30 years ago on an IBM PC clone.

> > (You never said you were talking about a console.)  We currently don't
> > expose console cursor control to Lisp.  Should be easy to add that, if
> > needed.
> 
> I see.
> 
> cursor-type does not say it does not work on the console.
> 
> I think not all values are doable and relevant.  t and nil should be
> enough.
> 
> Shell I open a bug report for this?

A feature request, yes.

> > I don't think we have such a hook, no.  But if we had it, using it
> > like you want would cause annoying flickering, because you'd
> > constantly redraw the image at very high frequency.
> 
> I do not think it would cause flickering on the console.  The only thing
> that normally "flickers" is the blinking cursor.  Emacs does not draw
> anything if nothing changed.

Emacs doesn't draw anything because it knows what's on the glass and
what _should_ be on the glass, and compares the two to decide what
should be redrawn.  But you want to draw on parts of the screen behind
Emacs's back, so it won't know whether something changed.  And you
OTOH have no way of knowing where Emacs have redrawn something during
the last redisplay cycle.  So you will have to redraw your images each
time Emacs finishes a redisplay cycle, regardless of whether it
changed anything on the glass in the area of an image.  These
redisplay cycles happen at very high frequency -- usually, Emacs
enters redisplay, quickly decides what needs to be redrawn, and then
does whatever has to be done and exits redisplay.  It tries very hard
not to redraw the parts that haven't changed, and that is why there's
no flickering.  But I don't see how you could prevent the flickering
of those add-on images, because without knowing which parts of the
screen changed, you will have to redraw them very frequently.

> >> Is there such hook?  If not, would it be difficult to add it?
> >
> > It won't be hard to add it, but I'm not sure we want to add such a
> > hook, see above.
> 
> That is a shame.  If there was a hook to detect when Emacs has drawn
> something on the console, I could plug image redrawing there.
> 
> If I wanted to add it, where in the code should I look?

In update_frame and its subroutines.  But my recommendation is not use
such a kludgey approach.