bug#41220: 26.3; Repeated yielding in worker thread insufficient to process input events in the main thread

[Alexander Miller]

This report is based on the discussion in
https://lists.gnu.org/archive/html/emacs-devel/2020-05/msg01566.html.

In this thread we were thinking up ways how to make Elisp
asynchronous with the help of threads - to do heavy lifting in a worker
thread, but yield for user interactions. Part of that was a benchmark
for how well persistently yielding in the worker thread after every
finished work-unit would perform.

My conclusion was that it doesn't, input was ignored most of the
time. Stefan thought that it *should* work, and asked me to make a bug
report, so here it is.

I have modified my benchmark to work with emacs -q. It will set up a
worker thread, a queue, a timer to check the queue and wake up the
worker, as well as a long CPU consuming task that emulates the kind of
work I would want to use threads for in a real-world scenario.

The task needs close to half a minute on my system. While it runs
navigation with C-n/p is barely possible.

Here is the full code:

(defconst worker-mutex (make-mutex "*WORKER MUTEX*"))
(defconst worker-cond-var (make-condition-variable worker-mutex))
(defvar worker-queue (list))

(setf
 worker
 (make-thread
  (lambda ()
    (while t
      (while worker-queue
        (let* ((work-unit (pop worker-queue))
               (fn (car work-unit))
               (args (cdr work-unit)))
          (apply fn args)
          (thread-yield)))
      (message "Worker Goes To Sleep")
      (with-mutex worker-mutex
        (condition-wait worker-cond-var))))
  "*WORKER*"))

(setf worker-timer
      (run-with-idle-timer
       1 t (lambda ()
             (when worker-queue
               (with-mutex worker-mutex
                 (condition-notify worker-cond-var))))))

(defun get-the-things ()
  (let ((run-start (float-time))
        (things))
    (dotimes (_ 100000)
      (let* ((status (prog1 (random 4) (dotimes (_ 1000) (random))))
             (thing (pcase status
                      (0 'a)
                      (1 'b)
                      (2 'c)
                      (3 'd))))
        (push thing things))
      (thread-yield))
    (message "Run Time %ss" (- (float-time) run-start))))

(push (list #'get-the-things) worker-queue)

In GNU Emacs 26.3 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.10)
of 2019-08-29 built on juergen
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Manjaro Linux

Recent messages:
worker-cond-var
worker-queue
#<thread *WORKER*>
Worker Goes To Sleep
[nil 0 1 0 t (closure (t) nil (if worker-queue (progn (let ((mutex
worker-mutex)) (mutex-lock mutex) (unwind-protect (progn
(condition-notify worker-cond-var)) (mutex-unlock mutex)))))) nil idle 0]
get-the-things
((get-the-things))
Auto-saving...
Run Time 90.80384421348572s
Worker Goes To Sleep

Configured using:
'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
--localstatedir=/var --with-x-toolkit=gtk3 --with-xft --with-modules
'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt'
CPPFLAGS=-D_FORTIFY_SOURCE=2
LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS GLIB
NOTIFY ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD LCMS2

Important settings:
value of $LC_MONETARY: de_DE.UTF-8
value of $LC_NUMERIC: de_DE.UTF-8
value of $LC_TIME: de_DE.UTF-8
value of $LANG: en_GB.utf8
locale-coding-system: utf-8

Major mode: Emacs-Lisp

Minor modes in effect:
ivy-posframe-mode: t
diff-auto-refine-mode: t
rainbow-delimiters-mode: t
rainbow-mode: t
display-line-numbers-mode: t
helm-mode: t
async-bytecomp-package-mode: t
helm--remap-mouse-mode: t
projectile-mode: t
global-company-mode: t
company-mode: t
global-evil-vimish-fold-mode: t
evil-vimish-fold-mode: t
vimish-fold-mode: t
show-smartparens-global-mode: t
show-smartparens-mode: t
smartparens-global-mode: t
smartparens-mode: t
org-super-agenda-mode: t
treemacs-icons-dired-mode: t
treemacs-filewatch-mode: t
treemacs-follow-mode: t
treemacs-git-mode: deferred
treemacs-fringe-indicator-mode: t
gcmh-mode: t
framey-mode: t
purpose-mode: t
shackle-mode: t
winum-mode: t
eyebrowse-mode: t
global-subword-mode: t
subword-mode: t
global-evil-surround-mode: t
evil-surround-mode: t
evil-lion-mode: t
evil-goggles-mode: t
global-undo-tree-mode: t
undo-tree-mode: t
shell-dirtrack-mode: t
evil-mode: t
evil-local-mode: t
tooltip-mode: t
global-eldoc-mode: t
eldoc-mode: t
electric-indent-mode: t
mouse-wheel-mode: t
prettify-symbols-mode: t
file-name-shadow-mode: t
global-font-lock-mode: t
font-lock-mode: t
auto-composition-mode: t
auto-encryption-mode: t
auto-compression-mode: t
line-number-mode: t
transient-mark-mode: t

Load-path shadows:
/usr/share/emacs/26.3/lisp/emacs-lisp/let-alist hides
/home/am/.emacs.d/straight/build/let-alist/let-alist
/usr/share/emacs/site-lisp/mu4e/mu4e-headers hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-headers
/usr/share/emacs/site-lisp/mu4e/mu4e-draft hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-draft
/usr/share/emacs/site-lisp/mu4e/mu4e-lists hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-lists
/usr/share/emacs/site-lisp/mu4e/mu4e-proc hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-proc
/usr/share/emacs/site-lisp/mu4e/org-mu4e hides
/home/am/.emacs.d/straight/build/mu4e/org-mu4e
/usr/share/emacs/site-lisp/mu4e/mu4e-message hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-message
/usr/share/emacs/site-lisp/mu4e/mu4e-compose hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-compose
/usr/share/emacs/site-lisp/mu4e/mu4e-speedbar hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-speedbar
/usr/share/emacs/site-lisp/mu4e/mu4e-vars hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-vars
/usr/share/emacs/site-lisp/mu4e/mu4e-org hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-org
/usr/share/emacs/site-lisp/mu4e/mu4e-mark hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-mark
/usr/share/emacs/site-lisp/mu4e/mu4e-context hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-context
/usr/share/emacs/site-lisp/mu4e/mu4e-view hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-view
/usr/share/emacs/site-lisp/mu4e/mu4e-utils hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-utils
/usr/share/emacs/site-lisp/mu4e/mu4e-contrib hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-contrib
/usr/share/emacs/site-lisp/mu4e/mu4e-actions hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-actions
/usr/share/emacs/site-lisp/mu4e/mu4e hides
/home/am/.emacs.d/straight/build/mu4e/mu4e
/usr/share/emacs/site-lisp/mu4e/mu4e-main hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-main
/usr/share/emacs/site-lisp/mu4e/mu4e-icalendar hides
/home/am/.emacs.d/straight/build/mu4e/mu4e-icalendar
~/Documents/git/treemacs/src/elisp/treemacs-interface hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-interface
~/Documents/git/treemacs/src/elisp/treemacs-rendering hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-rendering
~/Documents/git/treemacs/src/elisp/treemacs-dom hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-dom
~/Documents/git/treemacs/src/elisp/treemacs-tags hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-tags
~/Documents/git/treemacs/src/elisp/treemacs hides
/home/am/.emacs.d/straight/build/treemacs/treemacs
~/Documents/git/treemacs/src/elisp/treemacs-workspaces hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-workspaces
~/Documents/git/treemacs/src/elisp/treemacs-customization hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-customization
~/Documents/git/treemacs/src/elisp/treemacs-faces hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-faces
~/Documents/git/treemacs/src/elisp/treemacs-themes hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-themes
~/Documents/git/treemacs/src/elisp/treemacs-mouse-interface hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-mouse-interface
~/Documents/git/treemacs/src/elisp/treemacs-mode hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-mode
~/Documents/git/treemacs/src/elisp/treemacs-icons hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-icons
~/Documents/git/treemacs/src/elisp/treemacs-compatibility hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-compatibility
~/Documents/git/treemacs/src/elisp/treemacs-follow-mode hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-follow-mode
~/Documents/git/treemacs/src/elisp/treemacs-visuals hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-visuals
~/Documents/git/treemacs/src/elisp/treemacs-core-utils hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-core-utils
~/Documents/git/treemacs/src/elisp/treemacs-extensions hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-extensions
~/Documents/git/treemacs/src/elisp/treemacs-filewatch-mode hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-filewatch-mode
~/Documents/git/treemacs/src/elisp/treemacs-persistence hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-persistence
~/Documents/git/treemacs/src/elisp/treemacs-async hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-async
~/Documents/git/treemacs/src/elisp/treemacs-bookmarks hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-bookmarks
~/Documents/git/treemacs/src/elisp/treemacs-tag-follow-mode hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-tag-follow-mode
~/Documents/git/treemacs/src/elisp/treemacs-logging hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-logging
~/Documents/git/treemacs/src/elisp/treemacs-fringe-indicator hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-fringe-indicator
~/Documents/git/treemacs/src/elisp/treemacs-diagnostics hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-diagnostics
~/Documents/git/treemacs/src/elisp/treemacs-macros hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-macros
~/Documents/git/treemacs/src/elisp/treemacs-scope hides
/home/am/.emacs.d/straight/build/treemacs/treemacs-scope

Features:
(shadow sort mail-extr emacsbug cl-print debug benchmark eros
company-quickhelp pos-tip ivy-posframe posframe swiper ivy delsel
ivy-overlay colir help-fns radix-tree org-clock holidays hol-loaddefs
diary-lib diary-loaddefs cal-iso smartparens-javascript js
smartparens-html sgml-mode cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs smartparens-python python
tramp-sh tramp tramp-compat tramp-loaddefs trampver ucs-normalize json
map fish-mode conf-mode i3wm-config-mode writeroom-mode
visual-fill-column autorevert org-indent disp-table ol-eww eww mm-url
ol-rmail ol-mhe ol-irc ol-info ol-gnus nnir gnus-sum gnus-group
gnus-undo gnus-start gnus-cloud nnimap nnmail mail-source tls gnutls
utf7 netrc nnoo parse-time gnus-spec gnus-int gnus-range gnus-win gnus
nnheader ol-docview ol-bibtex bibtex ol-bbdb ol-w3m elfeed-org
elfeed-show elfeed-search wid-edit treemacs-bookmarks bookmark
elfeed-csv elfeed elfeed-curl url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf elfeed-log elfeed-db
elfeed-lib url-queue xml-query mailcap vc-mtn vc-hg vc-git diff-mode
vc-bzr vc-src vc-sccs vc-svn vc-cvs vc-rcs vc vc-dispatcher
rainbow-delimiters rainbow-mode xterm-color display-line-numbers
elec-pair winner helm-command helm-elisp helm-eval edebug helm-info info
shr-color color url-util url-parse url-vars shr svg dom face-remap pp
helm-mode helm-projectile helm-files helm-tags helm-buffers helm-occur
helm-grep helm-regexp helm-utils helm-locate helm-help helm-types
framey-helm helm-config async-bytecomp helm helm-source helm-multi-match
helm-lib async view treemacs-projectile projectile grep compile ibuf-ext
ibuffer ibuffer-loaddefs mu4e-alert time alert log4e notifications dbus
company-keywords company-dabbrev-code company-dabbrev company-yasnippet
company-files company-capf company server evil-vimish-fold vimish-fold
smartparens-config smartparens-org smartparens-text paren smartparens
xml gntp org-mu4e mu4e desktop frameset mu4e-org german-holidays
org-super-agenda ts org-habit org-element avl-tree generator org-agenda
org-refile org-bullets org ob ob-tangle ob-ref ob-lob ob-table ob-exp
org-macro org-footnote org-src ob-comint org-pcomplete org-list
org-faces org-entities noutline outline org-version ob-emacs-lisp
ob-core ob-eval org-table ol org-keys org-compat org-macs org-loaddefs
find-func mu4e-main mu4e-view cal-menu calendar cal-loaddefs browse-url
mu4e-headers mu4e-compose mu4e-context mu4e-draft mu4e-actions ido
rfc2368 smtpmail auth-source sendmail mu4e-mark mu4e-message flow-fill
mu4e-proc mu4e-utils doc-view jka-compr mu4e-lists mu4e-vars message rmc
puny seq treemacs-icons-dired treemacs-evil treemacs
treemacs-header-line treemacs-compatibility treemacs-mode
treemacs-interface treemacs-extensions treemacs-persistence
treemacs-mouse-interface treemacs-tag-follow-mode
treemacs-filewatch-mode treemacs-tags imenu xref project filenotify
treemacs-follow-mode treemacs-rendering treemacs-async
treemacs-workspaces treemacs-dom treemacs-visuals
treemacs-fringe-indicator treemacs-scope treemacs-faces treemacs-icons
treemacs-themes treemacs-core-utils pfuture ace-window avy hl-line
treemacs-macros treemacs-logging treemacs-customization easy-mmode
dired+ image-dired image-mode image-file dired-x dired-aux dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg
epg-config gnus-util rmail rmail-loaddefs mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr
mailabbrev mail-utils gmm-utils mailheader mu4e-meta gcmh hydra lv pcase
doom-modeline doom-modeline-segments doom-modeline-env
doom-modeline-core shrink-path rx f all-the-icons all-the-icons-faces
data-material data-weathericons data-octicons data-fileicons
data-faicons data-alltheicons memoize subr-x framey inline ht s
window-purpose window-purpose-fixes window-purpose-prefix-overload
window-purpose-switch let-alist window-purpose-layout
window-purpose-core window-purpose-configuration eieio-compat eieio
byte-opt bytecomp byte-compile cconv window-purpose-utils shackle trace
cl-extra help-mode winum eyebrowse format-spec morning-star-theme
cap-words superword subword evil-surround evil-lion evil-goggles pulse
cl-seq evil evil-keybindings evil-integration undo-tree diff evil-maps
evil-commands reveal flyspell ispell evil-jumps evil-command-window
evil-types evil-search evil-ex shell pcomplete comint ansi-color
evil-macros evil-repeat evil-states evil-core advice evil-common
windmove thingatpt rect evil-digraphs evil-vars ring edmacro kmacro dash
yasnippet-autoloads yaml-mode-autoloads xterm-color-autoloads
wttrin-autoloads writeroom-mode-autoloads with-editor-autoloads
winum-autoloads window-purpose-autoloads eieio-core cl-macs gv
eieio-loaddefs cl-loaddefs cl-lib vterm-autoloads
visual-fill-column-autoloads vimish-fold-autoloads undo-tree-autoloads
ts-autoloads tridactylrc-mode-autoloads treepy-autoloads
treemacs-autoloads transient-autoloads swiper-autoloads
straight-autoloads spinner-autoloads smartparens-autoloads
shrink-path-autoloads shackle-autoloads s-autoloads
rainbow-mode-autoloads rainbow-delimiters-autoloads projectile-autoloads
pretty-hydra-autoloads posframe-autoloads pos-tip-autoloads
popup-autoloads pkg-info-autoloads pfuture-autoloads
persp-mode-autoloads org-super-agenda-autoloads org-bullets-autoloads
org-autoloads multi-compile-autoloads mu4e-alert-autoloads
mu4e-autoloads morning-star-autoloads memoize-autoloads
markdown-mode-autoloads magit-autoloads macrostep-autoloads lv-autoloads
lsp-ui-autoloads lsp-treemacs-autoloads lsp-mode-autoloads
loop-autoloads log4e-autoloads let-alist-autoloads ledger-mode-autoloads
ivy-posframe-autoloads ivy-autoloads imenu-list-autoloads
i3wm-config-mode-autoloads hydra-autoloads ht-autoloads
helpful-autoloads helm-projectile-autoloads helm-org-autoloads
helm-easymenu easymenu helm-core-autoloads helm-ag-autoloads
helm-autoloads goto-chg-autoloads gntp-autoloads git-commit-autoloads
ghub-autoloads german-holidays-autoloads gcmh-autoloads framey-autoloads
forge-autoloads flycheck-pos-tip-autoloads flycheck-autoloads
fish-mode-autoloads fill-column-indicator-autoloads f-autoloads
eyebrowse-autoloads expand-region-autoloads evil-vimish-fold-autoloads
evil-surround-autoloads evil-nerd-commenter-autoloads
evil-magit-autoloads evil-lion-autoloads evil-ledger-autoloads
evil-goggles-autoloads evil-exchange-autoloads evil-collection-autoloads
evil-autoloads eros-autoloads epl-autoloads emacsql-sqlite-autoloads
emacsql-autoloads elisp-refs-autoloads elfeed-org-autoloads
elfeed-autoloads doom-modeline-autoloads doct-autoloads dired+-autoloads
dash-functional-autoloads dash-autoloads company-shell-autoloads
company-quickhelp-autoloads company-makefile-autoloads
company-box-autoloads company-autoloads closql-autoloads
buttercup-autoloads avy-autoloads async-autoloads anzu-autoloads
annalist-autoloads all-the-icons-autoloads alert-autoloads
ace-window-autoloads time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel term/x-win x-win
term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads dbusbind inotify lcms2 dynamic-setting
system-font-setting font-render-setting move-toolbar gtk x-toolkit x
multi-tty make-network-process emacs)

Memory information:
((conses 16 1134735 336269)
(symbols 48 62609 1)
(miscs 40 669 2944)
(strings 32 246316 89817)
(string-bytes 1 7685472)
(vectors 16 113372)
(vector-slots 8 2276151 279318)
(floats 8 6416 2552)
(intervals 56 38016 2345)
(buffers 992 44))