bug#40864: 27.0.91; Infinite loop in composing ligatures in overlays

[Gerry Agbobada]

First : an investigation around this bug has been led in emacs-devel,
and it turns out commit fe903c5ab7354 on master branch fixes it. I am
reporting the bug for other emacs 27 users who might find this while
looking for a solution.

It is currently impossible to have an overlay with ligatures when trying
to have proper ligatures in `composition-function-table' :
;;; ligature-bug-light.el --- Reproduce an autocomposition bug -*-
lexical-binding: t; -*-
;;; Commentary:
;;
;; Reproducer for the composition table bug
;;
;;;; Method
;;
;; Evaluate the file line by line
;; Expected : "=> testing overlays" is displayed at some point and disappear
;; Actual : evaluting the 'after-string overlay-put triggers infinite
;; looping, trying to "shape unibyte text",
;; which happens to be an empty string (from my earlier debugging)
;;
;;; Code:

(defvar ligature-font-with-big-arrow "Fira Code 11"
"A valid font name for `set-frame-font'. A font with a => ligature is
necessary to trigger the bug.")
(defvar bugged-overlay nil "Bugged overlay trying to compose an empty string.")

;;;; Setup
;; Requiring composite
(require 'composite)

;; Shaping => into a ligature
(set-char-table-range composition-function-table ?= '(["=>" 0
font-shape-gstring]))

;; Setting the font
;; =>
;; At this line
(set-frame-font ligature-font-with-big-arrow nil t)
;; Now you should see ligatures for =>
;; =>

;;;; Enabling the bug
(setq bugged-overlay (make-overlay (point-at-bol 4) (point-at-eol 4)
(current-buffer)))
(overlay-put bugged-overlay 'category 'test-overlays)
;; C-x C-e at the end of this sexp will trigger infinite composition
(overlay-put bugged-overlay 'after-string "=> testing overlays")

(delete-overlay bugged-overlay) ;; cleanup in case you have a working version

(provide 'ligature-bug-light)
;;; ligature-bug-light.el ends here

In GNU Emacs 27.0.91 (build 3, x86_64-pc-linux-gnu, GTK+ Version 3.22.30)
of 2020-04-26 built on LPPS0066
Repository revision: 68f4a740a13ee6a1d98079ef655dd42924d24d41
Repository branch: emacs-27
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
System Description: Ubuntu 18.04.4 LTS

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
ligature-bug-light.el has auto save data; consider M-x recover-this-file

Configured using:
'configure --with-libzstd-from-internet --with-libb2-from-internet'

Configured features:
XPM JPEG TIFF GIF PNG SOUND DBUS GSETTINGS GLIB NOTIFY INOTIFY GNUTLS
LIBXML2 FREETYPE HARFBUZZ XFT ZLIB TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM
MODULES THREADS JSON PDUMPER GMP

Important settings:
value of $LC_MONETARY: fr_FR.UTF-8
value of $LC_NUMERIC: fr_FR.UTF-8
value of $LC_TIME: fr_FR.UTF-8
value of $LANG: en_US.UTF-8
locale-coding-system: utf-8-unix

Major mode: Emacs-Lisp

Minor modes in effect:
tooltip-mode: t
global-eldoc-mode: t
eldoc-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
menu-bar-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
blink-cursor-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
line-number-mode: t
transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
format-spec rfc822 mml easymenu mml-sec password-cache epa derived epg
epg-config gnus-util rmail rmail-loaddefs text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 44895 10652)
(symbols 48 6006 1)
(strings 32 15487 1594)
(string-bytes 1 507021)
(vectors 16 10026)
(vector-slots 8 130535 10300)
(floats 8 23 69)
(intervals 56 294 0)
(buffers 1000 13))