bug#50629: 28.0.50; hard to debug an uncaught error with ert

[Mike Kupfer]

(This is similar to bug#49805, but I'm filing a new bug in case this
issue needs a different resolution than #49805.)

Consider the following stripped-down test case:

$ emacs -Q -batch -l ert -eval '(ert-deftest f () (defun foo () (bar)) (defun 
bar () (error "bar")) (should (equal 42 (foo))))' -f 
ert-run-tests-batch-and-exit

It produces

-----8<-----8<-----
Running 1 tests (2021-09-16 15:55:17-0700)
Test f backtrace:
  signal(error ("bar"))
  apply(signal (error ("bar")))
  (setq value-2 (apply fn-0 args-1))
  (unwind-protect (setq value-2 (apply fn-0 args-1)) (setq form-descri
  (if (unwind-protect (setq value-2 (apply fn-0 args-1)) (setq form-de
  (let (form-description-4) (if (unwind-protect (setq value-2 (apply f
  (let ((value-2 'ert-form-evaluation-aborted-3)) (let (form-descripti
  (let* ((fn-0 (function equal)) (args-1 (condition-case err (let ((si
  (lambda nil (defalias 'foo (function (lambda nil (bar)))) (defalias 
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name f :documentation nil :body (lambda ni
  ert-run-or-rerun-test(#s(ert--stats :selector t :tests [#s(ert-test 
  ert-run-tests(t #f(compiled-function (event-type &rest event-args) #
  ert-run-tests-batch(nil)
  ert-run-tests-batch-and-exit()
  command-line-1(("-l" "ert" "-eval" "(ert-deftest f () (defun foo () 
  command-line()
  normal-top-level()
Test f condition:
    (error "bar")
   FAILED  1/1  f

Ran 1 tests, 0 results as expected, 1 unexpected (2021-09-16 15:55:17-0700)

1 unexpected results:
   FAILED  f
----->8----->8-----

Knowing that error "bar" was raised is helpful, but in a more realistic
test scenario, it can take some time to figure out where the error was
raised.  Setting debug-on-error to t doesn't help (does ert hijack the
normal error handling mechanisms?).  The only way I know of to track it
down is to run the test multiple times, and use #'message at key points
in the code to record execution progress.

I ran into this while running some (not-yet-integrated) ert tests for
MH-E.  The message in the *ert* buffer looked like

  F mh-folder-completion-function-10-plus-slash-abs-folder
      Test ‘mh-folder-completion-function’ with ‘+/abso-folder’.
      (args-out-of-range "folder/bar" 44 nil)

The backtrace produced by ert was over 1MB long, and I was unable to use
it to determine where the error was coming from.

Here are two suggestions for how to make this better.

1. When ert detects an uncaught error, capture the backtrace at that
time and include the backtrace in the results.

2. Provide a way for an interactive user to disable whatever ert is
doing that lets it capture errors and drive on.  Instead, just let the
error be raised, and let the user use the Lisp debugger to investigate.


In GNU Emacs 28.0.50 (build 2, x86_64-pc-linux-gnu, X toolkit, cairo version 
1.16.0, Xaw scroll bars)
 of 2021-09-02 built on deb10
Repository revision: 6885c62a104dcf52378860d8f951dbdce52b73c5
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12004000
System Description: Debian GNU/Linux 10 (buster)

Configured using:
 'configure --prefix=/usr/new'

Configured features:
CAIRO FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG LIBSELINUX
LIBXML2 MODULES NOTIFY INOTIFY PDUMPER PNG SECCOMP SOUND THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM LUCID ZLIB

Important settings:
  value of $LC_TIME: C
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: ERT-Results

Minor modes in effect:
  shell-dirtrack-mode: t
  delete-selection-mode: t
  global-eldoc-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  buffer-read-only: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow mh-mime mh-identity mh-tool-bar mh-letter mh-show goto-addr
gnus-cite gnus-art mm-uu mml2015 gnus-sum shr kinsoku svg dom browse-url
url url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util url-parse url-vars gnus-group gnus-undo gnus-start
gnus-dbus dbus xml gnus-cloud nnimap nnmail mail-source utf7 netrc nnoo
parse-time iso8601 gnus-spec gnus-int gnus-win gnus-range mh-xface
mh-comp mh-scan mh-gnus mm-view mml-smime smime dig mailcap emacsbug
eieio-opt speedbar ezimage dframe shortdoc apropos thingatpt help-fns
radix-tree mule-util jka-compr info misearch multi-isearch ediff-vers
ediff ediff-merg ediff-mult ediff-wind ediff-diff ediff-help ediff-init
ediff-util smerge-mode diff whitespace cl-extra gnus nnheader wid-edit
mdk-mail smtpmail sendmail message rmc puny rfc822 mml mml-sec epa
derived epg rfc6068 epg-config gnus-util rmail rmail-loaddefs
auth-source eieio eieio-core eieio-loaddefs password-cache json map
text-property-search time-date subr-x mm-decode mm-bodies mm-encode
mail-parse rfc2231 rfc2047 rfc2045 mm-util ietf-drums mail-prsvr
mail-utils gmm-utils mailheader cl-macs mh-utils mh-e mh-compat
mailabbrev mh-buffers mh-loaddefs ert pp ewoc debug backtrace help-mode
find-func vc-git diff-mode bug-reference dired-aux dired-x seq gv dired
dired-loaddefs server noutline outline easy-mmode cc-mode cc-fonts
cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine cc-vars cc-defs
byte-opt bytecomp byte-compile cconv shell pcomplete comint ansi-color
ring xcscope advice delsel cl-seq vc vc-dispatcher timeclock cl-loaddefs
cl-lib mdk-hacks iso-transl tooltip eldoc electric uniquify ediff-hook
vc-hooks lisp-float-type mwheel term/x-win x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode elisp-mode lisp-mode prog-mode register page
tab-bar menu-bar rfn-eshadow isearch easymenu timer select scroll-bar
mouse jit-lock font-lock syntax font-core term/tty-colors frame
minibuffer cl-generic cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite charscript charprop case-table epa-hook jka-cmpr-hook help
simple abbrev obarray cl-preloaded nadvice button loaddefs faces
cus-face macroexp files window text-properties overlay sha1 md5 base64
format env code-pages mule custom widget hashtable-print-readable
backquote threads inotify dynamic-setting system-font-setting
font-render-setting cairo x-toolkit x multi-tty make-network-process
emacs)

Memory information:
((conses 16 314657 10561)
 (symbols 48 22275 4)
 (strings 32 75491 1172)
 (string-bytes 1 2515695)
 (vectors 16 38901)
 (vector-slots 8 438662 8550)
 (floats 8 300 273)
 (intervals 56 27726 0)
 (buffers 992 25))