(almost a patch) Receiving more output from a Common Lisp evaluation in Org buffer

[akater]

* Summary
I have a patch that allows to put trace output and error output into
corresponding Org buffer. Current behaviour with outputs being splitted
so that they go to different buffers (Org buffer, REPL), is arbitrary
and inconvenient. The patch ensures backwards compatibility. I thus
believe merging the patch would be a clear improvement. I use the
proposed features daily (trace, mostly). Note however that the patch
can't be merged right away; see [[Caveats]].

Contents of output streams not specified in =:results= is always emitted
to REPL buffer.

* Examples
I'll give here some examples of what's possible with the patch applied
that is not possible currently.

** Output of ~time~
#+begin_src lisp :results trace
(time 0)
#+end_src

#+RESULTS:
: Evaluation took:
:   0.000 seconds of real time
:   0.000004 seconds of total run time (0.000004 user, 0.000000 system)
:   100.00% CPU
:   420 processor cycles
:   0 bytes consed
:   

** Traces
#+begin_src lisp :results trace :wrap example lisp
(defun memq (item list) (or (eq item (car list)) (memq item (cdr list))))
(trace memq)
(memq 'e '(a b c d e f))
#+end_src

#+RESULTS:
#+begin_example lisp
  0: (MEMQ E (B C D E F))
    1: (MEMQ E (C D E F))
      2: (MEMQ E (D E F))
        3: (MEMQ E (E F))
        3: MEMQ returned T
      2: MEMQ returned T
    1: MEMQ returned T
  0: MEMQ returned T
#+end_example

** Ignored errors
#+begin_src lisp :results errors :wrap example lisp
(ignore-errors (/ 1 0))
#+end_src

#+RESULTS:
#+begin_example lisp
; in: LET ((*DEFAULT-PATHNAME-DEFAULTS* #P"/home/akater/"))
;     (/ 1 0)
; 
; caught STYLE-WARNING:
;   Lisp error during constant folding:
;   arithmetic error DIVISION-BY-ZERO signalled
;   Operation was (/ 1 0).
; 
; compilation unit finished
;   caught 1 STYLE-WARNING condition
#+end_example

** A combo
#+begin_src lisp :results errors trace output :wrap example lisp
(progn (time 0) (ignore-errors (/ 1 0)) (princ "wow"))
#+end_src

#+RESULTS:
#+begin_example lisp
; in: LET ((*DEFAULT-PATHNAME-DEFAULTS* #P"/home/akater/"))
;     (/ 1 0)
; 
; caught STYLE-WARNING:
;   Lisp error during constant folding:
;   arithmetic error DIVISION-BY-ZERO signalled
;   Operation was (/ 1 0).
; 
; compilation unit finished
;   caught 1 STYLE-WARNING condition

Evaluation took:
  0.000 seconds of real time
  0.000004 seconds of total run time (0.000004 user, 0.000000 system)
  100.00% CPU
  420 processor cycles
  0 bytes consed
  

wow
#+end_example

* Caveats
This improvement requires a change in SLIME code. One of SLIME
maintaners agreed to merge it but SLIME and Org would need to
synchronise their updates, and I'm not sure how to approach this.

Corresponding patch to SLIME changes the interface of
~swank:eval-and-grab-output~. As implemented,
~swank:eval-and-grab-output~ makes its users choose between values and
output by means of ~car~ and ~cadr~; preserving the compatibility by
extending the current behaviour would make its users refer to various
outputs by their positions which is a very bad idea. I employed alist
instead. However, merging will break =ob-lisp= completely for those who
use vcs Org but a stable SLIME.

The change to SLIME may also affect ~org-babel-execute:clojure~ but I
have no idea why Clojure would use SLIME and whether it actually does
use it. Anyway, I will be able to provide a patch for it that would
ensure backwards compatibility.

* Some Details
** Syntax
At least one function needs to be changed, namely
~org-babel-execute:lisp~.

We preserve the current behaviour, i.e. =:results output= behaves as it
used to. Preserving compatibility requires patching
~org-babel-result-cond~ so that it does not try to vectorise output from
=:result trace=, =:result errors=.

We could avoid this by requiring the =output= parameter all the time, so
that user would have to write something like =:results trace output= or
=:results output trace standard=. While the former is neat, the latter
is not. Also, the notion “error output stream” makes sense not only for
Common Lisp. Thus, adding =trace= and =errors= exceptional cases to
~org-babel-result-cond~ looks acceptable to me.

** (suggestion) Multi-block return
Overall, I believe it would be best to embrace full potential of Org
features and implement multi-block results as default for Common Lisp
evaluation in Org, like in the following example:

#+begin_src lisp
(progn (time 0) (ignore-errors (/ 1 0)) (princ "wow") t)
#+end_src

#+RESULTS:
#+begin_values lisp
T
#+end_values
#+begin_errors lisp
; in: LET ((*DEFAULT-PATHNAME-DEFAULTS* #P"/home/akater/"))
;     (/ 1 0)
; 
; caught STYLE-WARNING:
;   Lisp error during constant folding:
;   arithmetic error DIVISION-BY-ZERO signalled
;   Operation was (/ 1 0).
; 
; compilation unit finished
;   caught 1 STYLE-WARNING condition
#+end_errors
#+begin_trace lisp
Evaluation took:
  0.000 seconds of real time
  0.000003 seconds of total run time (0.000003 user, 0.000000 system)
  100.00% CPU
  476 processor cycles
  0 bytes consed
  

#+end_trace
#+begin_output
wow
#+end_output

while simply hiding empty blocks, if any, for convenience.

Note that currently, when “output” is specified, “values” is simply
lost, and vice versa. Implementing multi-block results would fix this
shortcoming too.

However, I did not try to implement this yet.

* Conclusion
How do we sync with SLIME if you're OK with this? How do we treat the
case of vcs Org + stable SLIME?