bug#41959: Subject: 27.0.91; comint-prompt-regexp in Python mode is incorrect and causes completion failures

[Leo Alekseyev]

M-x run-python brings up a warning buffer with the following message:
Warning (python): Your ‘python-shell-interpreter’ doesn’t seem to
support readline, yet ‘python-shell-completion-native-enable’ was t
and "python" is not part of the
‘python-shell-completion-native-disabled-interpreters’ list.  Native
completions have been disabled locally.

This warning is unwarranted, because the python-shell-interpreter being
invoked in fact supports readline (as can be readily checked via
"import readline, rlcompleter").

Tracing through the code it appears that
python-shell-completion-native-setup erroneously returns nil, because
(python-shell-accept-process-output
              process python-shell-completion-native-try-output-timeout)
returns nil.

This, however, appears to be caused by the fact that the comint prompt
in Python mode is expected to look like ">>> ", which is a bad
assumption. When readline is run with "set show-mode-in-prompt on"
option, the comint prompt will be prefixed with the readline mode
indicator, so it will look like "@>>> ", or "(ins)>>> ", or "+>>> ",
depending on user's configuration (typically in ~/.inputrc).

In short, comint-prompt-regexp assumes a very specific environment, and
when comint prompt fails to match these expectations it causes unrelated
failures in e.g. completion functions.


In GNU Emacs 27.0.91 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.22.30)
 of 2020-06-08 built on meitner
Repository revision: 0260d2d2dbb2607e7310bdb518b7b6c0f58f5f98
Repository branch: emacs-27
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Ubuntu 20.04 LTS

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.
Waiting for git... [2 times]
Shell native completion is disabled, using fallback
V is undefined
Mark set [2 times]
Mark activated

Configured using:
 'configure --with-sound=alsa --without-gconf --with-mailutils
 --with-json --with-modules --prefix=/usr/local'

Configured features:
XPM JPEG TIFF GIF PNG RSVG SOUND DBUS GSETTINGS GLIB NOTIFY INOTIFY ACL
GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11 XDBE XIM MODULES THREADS LIBSYSTEMD JSON
PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Special

Minor modes in effect:
  shell-dirtrack-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/home/leo/.emacs.d.doom.27/.local/straight/build/let-alist/let-alist
hides /usr/local/share/emacs/27.0.91/lisp/emacs-lisp/let-alist

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg epg-config gnus-util rmail
rmail-loaddefs text-property-search mm-decode mm-bodies mm-encode
mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils warnings compile
cl-extra help-mode python tramp-sh tramp tramp-loaddefs trampver
tramp-integration files-x tramp-compat shell pcomplete parse-time
iso8601 time-date ls-lisp format-spec comint ring ansi-color finder-inf
package easymenu browse-url url-handlers url-parse auth-source cl-seq
eieio eieio-core cl-macs eieio-loaddefs password-cache json subr-x map
url-vars seq byte-opt gv bytecomp byte-compile cconv cl-loaddefs cl-lib
tooltip eldoc electric uniquify ediff-hook vc-hooks lisp-float-type
mwheel term/x-win x-win term/common-win x-dnd tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
timer select scroll-bar mouse jit-lock font-lock syntax facemenu
font-core term/tty-colors frame minibuffer cl-generic cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite charscript charprop
case-table epa-hook jka-cmpr-hook help simple abbrev obarray
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote threads dbusbind
inotify lcms2 dynamic-setting system-font-setting font-render-setting
move-toolbar gtk x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 80485 7435)
 (symbols 48 9871 1)
 (strings 32 29404 1906)
 (string-bytes 1 1053067)
 (vectors 16 15547)
 (vector-slots 8 194915 7402)
 (floats 8 58 12)
 (intervals 56 216 0)
 (buffers 1000 14))