Terminal or keyboard decoding system?

[Ravi R]

My attempt to use all the standard emacs modifiers (shift, control,
meta, alt, super, hyper) on *terminal* emacs has been fairly successful,
but it's not clear to me that my current approach of modifying
input-decode-map and key-translation-map is the correct one, as it seems
to have some pitfalls I do not understand. Please help me identify the
correct way to decode these terminal sequences.

My terminal emulator of choice, kitty, now provides an unambiguous
mechanism [1] for sending all the modifier keys listed above for any key
press. The protocol is straightfoward; every key (without exception) is
reported to the application in one of the following forms:
  CSI number ; modifier u
  CSI 1 ; modifier suffix
where spaces have been added above for clarity, but are not part of the
protocol.
  CSI: "\e[" (as an emacs string)
  number: decimal number in ASCII, e.g., two byte string "97" for ?a
  modifier: decimal number in ASCII representing a bit-field (see below)
  suffix: one character for some specific keys
The modifier decimal number minus one represents each modifier in one of
its bits, e.g., shift is bit LSB, alt is bit 1, etc.

My current approach [2] is to bind the CSI sequence to the following
function in input-decode-map:

(fset 'xterm-kitty--original-read-char-exclusive (symbol-function 
'read-char-exclusive))
(defun xterm-kitty--handle-escape-code (prompt)
  "Handle keycode using integer math; PROMPT is ignored."
  (let ((keycode 0)
        (modifiers 0)
        (suffix nil)
        (current-num 0)
        (e))
    (while (not suffix)
      (setq e (xterm-kitty--original-read-char-exclusive))
      (if (<= ?0 e ?9)
          (setq current-num (+ (* current-num 10) (- e ?0)))
        (if (eql e ?\;)
            (setq keycode current-num
                  current-num 0)
          (setq suffix e)
          (if (> keycode 0)
              (setq modifiers (1- current-num))
            (setq keycode current-num)))))
    ;; (message "Code: %d modifiers %d suffix: %s" keycode modifiers suffix)
    (xterm-kitty-decode-key-stroke keycode modifiers suffix)))

where  xterm-kitty-decode-key-stroke creates the appropriate emacs key
sequence, e.g., "\e[97;51u" goes to it as (97 51 u), which is then
mapped to (kbd "H-M-A-a") since 51-1 = 32(meta)+16(hyper)+2(alt). This
approach has two major pitfalls:
  1. read-char-exclusive can be advised by packages
  2. C-DEL has some issues
  3. Standard C-, M-, C-M- bindings cannot distinguish between shifted
     and non-shifted variants for characters: abcdefghjklnopqrstuvwxyz\
I don't understand the "C-DEL" issue at all, and work around the
shifted/non-shifted bindings via the following hacks:
  a. Mapping shifted to non-shifted versions in key-tranlation-map
  b. Add any non-shifted personal bindings (e.g., to "C-A" rather than
     "C-a") in local-function-key-map (really ugly)
The first issue of read-char-exclusive advise by packages cannot be
solved at all, as far as I can tell.

Questions:

1. Is modifying input-decode-map and key-translation-map the right
   approach? Or should this be done using a coding system, e.g.,
   keyboard-coding-system or terminal-coding-system? If so, can CCL or
   equivalent be used to translate from the keyboard protocol to emacs
   key representations? As far as I can tell, CCL programs transform a
   byte sequence to a different byte sequence; how can they be used to
   produce emacs key representations? I'm happy to write C code to
   implement this new coding system, if that's the best approach. Or is
   there a function at a lower level than read-char-exclusive that
   should be used?

2. Emacs key representations seem to come in two forms: a 28-bit integer
   (e.g., "H-M-A-a" is (1<<24)+(1<<27)+(1<<22)+97) and a symbol list
   (e.g., "H-<f1>" is 'H-f1), both of which are then stored in a vector.
   To be able to disambiguate between shifted versions and non-shifted
   versions, I ended up computing [3] the values directly for the
   integer representation, since event-convert-list (used internally by
   define-key) strips off shift modifiers for alphabetic characters in C
   code, and provides no way to work around it. However, "C-a" and
   "C-S-a" seem to be work perfectly fine if the integer value is
   computed explicitly; is this expected, or are there other minefields
   that I haven't encountered?

3. When allowing mouse focus-in/out and or bracketed paste events,
   occasionally, they seem to interleave with keyboard input read using
   read-char-exclusive. This problem has been very hard to reproduce
   deterministically, but happens once or twice daily. How should one
   debug such issues?

Even with all of these issues, the ability to use all 6 modifier keys on
a 24-bit color terminal with multiple real frames (rather than just one
frame) over chained SSH connections has been fantastic! For my use
cases, font sizes/shapes and image display seem to be the only missing
features compared to graphical emacs. Implementing meta/hyper support in
kitty (after giving up on a more famous terminal emulator), determining
heuristics for detecting modifiers in XKB on Wayland (which does not
natively provide a way to query them), and modifying XKB system
configuration on Wayland to support all these modifiers without breaking
other applications, has been quite a journey to getting a terminal emacs
experience on par with graphical emacs.

Regards,
Ravi

[1] 
https://sw.kovidgoyal.net/kitty/keyboard-protocol/#report-all-keys-as-escape-codes
[2] 
http://cgit.lexarcana.com/cgit.cgi/dotemacs/tree/lisp/term/xterm-kitty.el#n390
[3] 
http://cgit.lexarcana.com/cgit.cgi/dotemacs/tree/lisp/term/xterm-kitty.el#n351