bug#70000: 29.2; Grapheme handling incorrect

[Eli Zaretskii]

> From: jman <jman@city17.xyz>
> Cc: 70000@debbugs.gnu.org, Phillip Susi <phill@thesusis.net>
> Date: Thu, 26 Jun 2025 23:55:33 +0200
> 
> 
> > The bottom line is what I said at the beginning: we need some protocol
> > by which a terminal emulator could be queried about whether it
> > supports character composition, and if so, what is the screen width of
> > a given sequence of codepoints that will be composed, without actually
> > displaying them.  Better yet, some standard table of such widths could
> > be accepted by complying terminal emulators, and then Emacs could use
> > such a table to know the width in advance (similarly to how it knows
> > that from the Unicode data files).
> > 
> > Until such protocols or tables exist, Emacs will be unable to produce
> > correct display on these terminal emulators.
> 
> Eli, Phillip, I think today such protocol exists.
> 
> The author of the terminal emulator Kitty implemented a text-sizing 
> protocol[0] that tells any text 
> editor how to treat graphemes and especially grapheme clusters. For example 
> if I paste the emoji of 
> the "writing hand"[1] (which is composed by 2 graphemes U+270D and U+FE0F) 
> into the buffer I am 
> writing into, Emacs does not understand how to display it, just as other TTY 
> text editors don't (I 
> also tried nano). Note that this happens only when using `emacs 
> --no-window-system`. When using 
> `emacs-pgtk` all is good (as Eli points out, Harfbuzz does the work to treat 
> grapheme correctly).
> 
> According to the author of Kitty[2], the mode 2027 mentioned in this thread 
> was not a suitable 
> solution.
> 
> I am curious to hear your thoughts about this, if this could be a way out to 
> fix the issue. I 
> realize that this is a protocol developed by a single person and not a 
> committee with wide industry 
> adoption. but as far as I know it's the only working solution to the problem.
> 
> I am using Emacs 30.1 and I am experiencing the same problem reported in this 
> thread since basically 
> ever.

Thanks.

The protocol described in

  https://sw.kovidgoyal.net/kitty/text-sizing-protocol/

if we decide to implement it in Emacs, will need some non-trivial
changes in how Emacs currently accounts for character width on
display.  That is because this protocol does NOT allow to query the
terminal about the display width of a string of characters.  Instead,
it allows a program running on the terminal to instruct the terminal
about the display width it expects to get, and the terminal needs to
obey.  What this means for Emacs is that we will have to add code
which will determine the expected display width of each composed
sequence of characters.  By contrast, what we have now is that we
expect the display backend to tell us the display width.

This is important because Emacs has code which performs layout
calculations by using the display code without actually displaying
anything.  Cursor movement commands in Emacs, and many places within
the display engine, use these capabilities.  When this code runs, it
needs some way of computing the display width of each glyph that will
or would be displayed.  If we need to compute that ourselves, we will
need to add such a code, which currently doesn't exist.

Beyond that, there's the issue of how widely will this protocol be
supported by terminal emulators other than kitty, and what should
Emacs do when it runs on a terminal which doesn't support this.