[Orgmode] [Babel] Executing sh code within sessions

[Sébastien Vauban]

Hi Eric,

As requested, I'm here emphasizing the important part of this post:

When executing sh code block within a session, it never terminates, echoing...

: executing Sh code block...

in the minibuffer.

The =sva= shell buffer contains:

#+begin_src sh
echo In $(pwd):

/address@hidden:...ples/Org-scraps 0$ In 
/cygdrive/c/home/sva/Examples/Org-scraps:
/address@hidden:...ples/Org-scraps 0$ echo In $(pwd):
In /cygdrive/c/home/sva/Examples/Org-scraps:
/address@hidden:...ples/Org-scraps 0$ ls *.org
Agenda-Sorting-Strategy.org
Clock-Report.org
org-beamer-fpu-rules.org
org-hist.org
/address@hidden:...ples/Org-scraps 0$ echo 'org_babel_sh_eoe'
org_babel_sh_eoe
/address@hidden:...ples/Org-scraps 0$
#+end_src

Does that ring a bell to you?

Details below, if needed.

Best regards,
  Seb

Sébastien Vauban wrote:
> #+TITLE:     Executing sh-code within a session
> #+AUTHOR:    Seb Vauban
> #+LANGUAGE:  en_US
>
> * Executing sh-code within a session
>
> ** Context
>
> - git pull done this morning (2010-11-04 Thu).
>
> - EmacsW32 + Cygwin bash as inferior shell
>
>   #+begin_src emacs-lisp
>   (setq shell-file-name "bash")
>   (setenv "SHELL" "bash")
>   (setq shell-command-switch "-c")
>   (setq w32-quote-process-args t)
>   (setq explicit-shell-file-name "bash")
>   #+end_src
>
> ** First time
>
> The first time I execute the following:
>
> #+begin_src sh :session sva
> echo In $(pwd):
> ls *.org
> #+end_src
>
> I get the following error:
>
> #+begin_src emacs-lisp
> Debugger entered--Lisp error: (error "Marker does not point anywhere")
>   ansi-color-apply-on-region(#<marker in no buffer> #<marker at 17 in sva>)
>   ansi-color-process-output("")
>   run-hook-with-args(ansi-color-process-output "")
>   comint-send-input(nil t)
>   (lambda (line) (insert line) (comint-send-input nil t) (sleep-for 
> 0.25))("echo In $(pwd):")
>   mapc((lambda (line) (insert line) (comint-send-input nil t) (sleep-for 
> 0.25)) ("echo In $(pwd):" "ls *.org" "echo 'org_babel_sh_eoe'"))
>   (progn (goto-char (process-mark ...)) (let (... ...) (setq dangling-text 
> ...) (delete-region start end)) (mapc (lambda ... ... ... ...) (append ... 
> ...)) (while (progn ... ...) (accept-process-output ...)) (goto-char 
> (process-mark ...)) (insert dangling-text))
>   (unwind-protect (progn (goto-char ...) (let ... ... ...) (mapc ... ...) 
> (while ... ...) (goto-char ...) (insert dangling-text)) (remove-hook (quote 
> comint-output-filter-functions) (quote my-filt)))
>   (progn (fset (quote my-filt) (function* ...)) (add-hook (quote 
> comint-output-filter-functions) (quote my-filt)) (unwind-protect (progn ... 
> ... ... ... ... ...) (remove-hook ... ...)))
>   (unwind-protect (progn (fset ... ...) (add-hook ... ...) (unwind-protect 
> ... ...)) (if --cl-letf-bound-- (fset ... --cl-letf-save--) (fmakunbound 
> ...)))
>   (let* ((--cl-letf-bound-- ...) (--cl-letf-save-- ...)) (unwind-protect 
> (progn ... ... ...) (if --cl-letf-bound-- ... ...)))
>   (letf ((... ...)) (add-hook (quote comint-output-filter-functions) (quote 
> my-filt)) (unwind-protect (progn ... ... ... ... ... ...) (remove-hook ... 
> ...)))
>   (letf* ((... ...)) (add-hook (quote comint-output-filter-functions) (quote 
> my-filt)) (unwind-protect (progn ... ... ... ... ... ...) (remove-hook ... 
> ...)))
>   (flet ((my-filt ... ...)) (add-hook (quote comint-output-filter-functions) 
> (quote my-filt)) (unwind-protect (progn ... ... ... ... ... ...) (remove-hook 
> ... ...)))
>   (let ((string-buffer "") dangling-text raw) (flet (...) (add-hook ... ...) 
> (unwind-protect ... ...)) (if (and t body ...) (setq raw ...)) (split-string 
> string-buffer comint-prompt-regexp))
>   (progn (unless (org-babel-comint-buffer-livep session) (error "buffer %s 
> doesn't exist or has no process" session)) (set-buffer session) (let (... 
> dangling-text raw) (flet ... ... ...) (if ... ...) (split-string 
> string-buffer comint-prompt-regexp)))
>   (unwind-protect (progn (unless ... ...) (set-buffer session) (let ... ... 
> ... ...)) (set-match-data save-match-data-internal (quote evaporate)))
>   (let ((save-match-data-internal ...)) (unwind-protect (progn ... ... ...) 
> (set-match-data save-match-data-internal ...)))
>   (save-match-data (unless (org-babel-comint-buffer-livep session) (error 
> "buffer %s doesn't exist or has no process" session)) (set-buffer session) 
> (let (... dangling-text raw) (flet ... ... ...) (if ... ...) (split-string 
> string-buffer comint-prompt-regexp)))
>   (save-excursion (save-match-data (unless ... ...) (set-buffer session) (let 
> ... ... ... ...)))
>   (org-babel-comint-in-buffer session (let (... dangling-text raw) (flet ... 
> ... ...) (if ... ...) (split-string string-buffer comint-prompt-regexp)))
>   (org-babel-comint-with-output (session org-babel-sh-eoe-output t body) 
> (mapc (lambda ... ... ... ...) (append ... ...)))
>   (butlast (org-babel-comint-with-output (session org-babel-sh-eoe-output t 
> body) (mapc ... ...)) 2)
>   (mapcar (function org-babel-trim) (butlast (org-babel-comint-with-output 
> ... ...) 2))
>   (mapconcat (function org-babel-sh-strip-weird-long-prompt) (mapcar 
> (function org-babel-trim) (butlast ... 2)) "\n")
>   (if (not session) (org-babel-eval org-babel-sh-command (org-babel-trim 
> body)) (mapconcat (function org-babel-sh-strip-weird-long-prompt) (mapcar ... 
> ...) "\n"))
>   ((lambda (results) (when results ...)) (if (not session) (org-babel-eval 
> org-babel-sh-command ...) (mapconcat ... ... "\n")))
>   org-babel-sh-evaluate(#<buffer sva> "echo In $(pwd):\nls *.org\n" 
> ("replace"))
>   (org-babel-reassemble-table (org-babel-sh-evaluate session full-body 
> result-params) (org-babel-pick-name (cdr ...) (cdr ...)) (org-babel-pick-name 
> (cdr ...) (cdr ...)))
>   (let* ((session ...) (result-params ...) (full-body ...)) 
> (org-babel-reassemble-table (org-babel-sh-evaluate session full-body 
> result-params) (org-babel-pick-name ... ...) (org-babel-pick-name ... ...)))
>   org-babel-execute:sh("echo In $(pwd):\nls *.org\n" ((:colname-names) 
> (:rowname-names) (:result-params "replace") (:result-type . value) (:comments 
> . "") (:shebang . "") (:cache . "no") (:noweb . "no") (:tangle . "no") 
> (:exports . "code") (:results . "replace") (:session . "sva") (:hlines . 
> "no")))
>   funcall(org-babel-execute:sh "echo In $(pwd):\nls *.org\n" 
> ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . 
> value) (:comments . "") (:shebang . "") (:cache . "no") (:noweb . "no") 
> (:tangle . "no") (:exports . "code") (:results . "replace") (:session . 
> "sva") (:hlines . "no")))
>   ((lambda (result) (cond ... ... ...)) (funcall cmd body params))
>   (setq result ((lambda ... ...) (funcall cmd body params)))
>   (if (and (not arg) new-hash (equal new-hash old-hash)) (save-excursion 
> (goto-char ...) (end-of-line 1) (forward-char 1) (setq result ...) (message 
> ...) result) (message "executing %s code block%s..." (capitalize lang) (if 
> ... ... "")) (setq result (... ...)) (org-babel-insert-result result 
> result-params info new-hash indent lang) (run-hooks (quote 
> org-babel-after-execute-hook)) result)
>   (progn (fset (quote call-process-region) (function* ...)) (unless (fboundp 
> cmd) (error "No org-babel-execute function for %s!" lang)) (if (and ... 
> new-hash ...) (save-excursion ... ... ... ... ... result) (message "executing 
> %s code block%s..." ... ...) (setq result ...) (org-babel-insert-result 
> result result-params info new-hash indent lang) (run-hooks ...) result))
>   (unwind-protect (progn (fset ... ...) (unless ... ...) (if ... ... ... ... 
> ... ... result)) (if --cl-letf-bound-- (fset ... --cl-letf-save--) 
> (fmakunbound ...)))
>   (let* ((--cl-letf-bound-- ...) (--cl-letf-save-- ...)) (unwind-protect 
> (progn ... ... ...) (if --cl-letf-bound-- ... ...)))
>   (letf ((... ...)) (unless (fboundp cmd) (error "No org-babel-execute 
> function for %s!" lang)) (if (and ... new-hash ...) (save-excursion ... ... 
> ... ... ... result) (message "executing %s code block%s..." ... ...) (setq 
> result ...) (org-babel-insert-result result result-params info new-hash 
> indent lang) (run-hooks ...) result))
>   (letf* ((... ...)) (unless (fboundp cmd) (error "No org-babel-execute 
> function for %s!" lang)) (if (and ... new-hash ...) (save-excursion ... ... 
> ... ... ... result) (message "executing %s code block%s..." ... ...) (setq 
> result ...) (org-babel-insert-result result result-params info new-hash 
> indent lang) (run-hooks ...) result))
>   (flet ((call-process-region ... ...)) (unless (fboundp cmd) (error "No 
> org-babel-execute function for %s!" lang)) (if (and ... new-hash ...) 
> (save-excursion ... ... ... ... ... result) (message "executing %s code 
> block%s..." ... ...) (setq result ...) (org-babel-insert-result result 
> result-params info new-hash indent lang) (run-hooks ...) result))
>   (unwind-protect (flet (...) (unless ... ...) (if ... ... ... ... ... ... 
> result)) (setq call-process-region (quote 
> org-babel-call-process-region-original)))
>   (let* ((lang ...) (params ...) (cache\? ...) (result-params ...) (new-hash 
> ...) (old-hash ...) (body ...) (cmd ...) (dir ...) (default-directory ...) 
> (org-babel-call-process-region-original ...) (indent ...) result) 
> (unwind-protect (flet ... ... ...) (setq call-process-region ...)))
>   (progn (let* (... ... ... ... ... ... ... ... ... ... ... ... result) 
> (unwind-protect ... ...)))
>   (if (org-babel-confirm-evaluate info) (progn (let* ... ...)))
>   (when (org-babel-confirm-evaluate info) (let* (... ... ... ... ... ... ... 
> ... ... ... ... ... result) (unwind-protect ... ...)))
>   (let ((info ...)) (when (org-babel-confirm-evaluate info) (let* ... ...)))
>   org-babel-execute-src-block(nil ("sh" "echo In $(pwd):\nls *.org\n" 
> ((:colname-names) (:rowname-names) (:result-params "replace") (:result-type . 
> value) (:comments . "") (:shebang . "") (:cache . "no") (:noweb . "no") 
> (:tangle . "no") (:exports . "code") (:results . "replace") (:session . 
> "sva") (:hlines . "no")) "" nil 0))
>   (progn (org-babel-execute-src-block current-prefix-arg info) t)
>   (if info (progn (org-babel-execute-src-block current-prefix-arg info) t) 
> nil)
>   (let ((info ...)) (if info (progn ... t) nil))
>   org-babel-execute-src-block-maybe()
>   (or (org-babel-execute-src-block-maybe) (org-babel-lob-execute-maybe))
>   org-babel-execute-maybe()
>   call-interactively(org-babel-execute-maybe)
>   (cond ((commandp org-speed-command) (setq this-command org-speed-command) 
> (call-interactively org-speed-command)) ((functionp org-speed-command) 
> (funcall org-speed-command)) ((and org-speed-command ...) (eval 
> org-speed-command)) (t (let ... ...)))
>   (cond ((and org-use-speed-commands ...) (cond ... ... ... ...)) ((and ... 
> ... ... ...) (let ... ... ... ... ...)) (t (setq org-table-may-need-update t) 
> (self-insert-command N) (org-fix-tags-on-the-fly) (if 
> org-self-insert-cluster-for-undo ...)))
>   org-self-insert-command(1)
>   call-interactively(org-self-insert-command nil nil)
> #+end_src
>
> ** Next times
>
> The second time I execute it:
>
> #+begin_src sh :session sva
> echo In $(pwd):
> ls *.org
> #+end_src
>
> it never terminates, echoing:
>
> : executing Sh code block...
>
> in the minibuffer.
>
> The =sva= shell buffer contains:
>
> #+begin_src sh
> echo In $(pwd):
>
> /address@hidden:...ples/Org-scraps 0$ In 
> /cygdrive/c/home/sva/Examples/Org-scraps:
> /address@hidden:...ples/Org-scraps 0$ echo In $(pwd):
> In /cygdrive/c/home/sva/Examples/Org-scraps:
> /address@hidden:...ples/Org-scraps 0$ ls *.org
> Agenda-Sorting-Strategy.org
> Clock-Report.org
> org-beamer-fpu-rules.org
> org-hist.org
> /address@hidden:...ples/Org-scraps 0$ echo 'org_babel_sh_eoe'
> org_babel_sh_eoe
> /address@hidden:...ples/Org-scraps 0$
> #+end_src
>
> ** Value of my Bash prompt
>
> It now is defined as such:
>
> #+begin_src sh
> # my format of the prompt: mono-line and (often) colored
> function my_prompt_command ()
> {
>     RET=$?
>
>     # replace the $HOME prefix by ~ in the current directory
>     if [[ $HOME == ${PWD:0:${#HOME}} ]]; then
>         NEWPWD="~${PWD:${#HOME}}"
>     else
>         NEWPWD=$PWD
>     fi
>
>     # how many characters of the $PWD should be kept
>     local pwd_max_length=15
>
>     if [[ ${#NEWPWD} -gt $pwd_max_length ]]; then
>         local pwd_offset=$(( ${#NEWPWD} - $pwd_max_length ))
>         NEWPWD="...${NEWPWD:$pwd_offset:$pwd_max_length}"
>     fi
>
>     # prompt character
>     if [[ $(whoami) = "root" ]]; then
>         local PROMPTCHAR="#"
>     else
>         local PROMPTCHAR="$"
>     fi
>
>     setenv PS1 "/address@hidden:${NEWPWD} ${RET}${PROMPTCHAR} "
> }
>
> # shell prompt
> setenv PROMPT_COMMAND my_prompt_command
> #+end_src
>
> So, it now definitively is:
>
> - mono-line
> - uncolored
>
> That should be the basic requirements I needed to meet, right?
>
> ** Variable comint-prompt-regexp
>
> Value of the =comint-prompt-regexp= variable in my shell, used by Org-babel to
> digest output from the shell:
>
> #+begin_src emacs-lisp
> (switch-to-buffer "sva")
> (describe-variable 'comint-prompt-regexp)
> #+end_src
>
> #+results:
> #+begin_example
> comint-prompt-regexp is a variable defined in `comint.el'.
> Its value is 
> "^[^#$%>\n]*[#$%>] *"
> Local in buffer sva; global value is "^"
>
> Documentation:
> Regexp to recognize prompts in the inferior process.
> Defaults to "^", the null string at BOL.
>
> This variable is only used if the variable
> `comint-use-prompt-regexp' is non-nil.
>
> Good choices:
>   Canonical Lisp: "^[^> \n]*>+:? *" (Lucid, franz, kcl, T, cscheme, oaklisp)
>   Lucid Common Lisp: "^\\(>\\|\\(->\\)+\\) *"
>   franz: "^\\(->\\|<[0-9]*>:\\) *"
>   kcl: "^>+ *"
>   shell: "^[^#$%>\n]*[#$%>] *"
>   T: "^>+ *"
>
> This is a good thing to set in mode hooks.
> #+end_example
>
> This regexp should match my own prompt as it appears in your *shell*
> buffers locally (product of my personal =.bashrc=) configuration.
>
> It does, right?
>
> ** What about shell-prompt-pattern?
>
> Browsing along, it seems there is another interesting value to look at:
> =shell-prompt-pattern=.
>
> #+begin_src emacs-lisp
> (switch-to-buffer "sva")
> (describe-variable 'shell-prompt-pattern)
> #+end_src
>
> #+results:
> #+begin_example
> shell-prompt-pattern is a variable defined in `shell.el'.
> Its value is 
> "^[^#$%>\n]*[#$%>] *"
>
> Documentation:
> Regexp to match prompts in the inferior shell.
> Defaults to "^[^#$%>\n]*[#$%>] *", which works pretty well.
> This variable is used to initialize `comint-prompt-regexp' in the
> shell buffer.
>
> If `comint-use-prompt-regexp' is nil, then this variable is only used
> to determine paragraph boundaries.  See Info node `Shell Prompts' for
> how Shell mode treats paragraphs.
>
> The pattern should probably not match more than one line.  If it does,
> Shell mode may become confused trying to distinguish prompt from input
> on lines which don't start with a prompt.
>
> This is a fine thing to set in your `.emacs' file.
>
> You can customize this variable.
> #+end_example
>
> I don't know if this was important, but -- as you can see -- I've set it to
> the same value as =comint-prompt-regexp=.
>
> ** Back to comint's prompt regexp!
>
> #+begin_src emacs-lisp
> (switch-to-buffer "sva")
> (describe-variable 'comint-use-prompt-regexp)
> #+end_src
>
> #+results:
> #+begin_example
> comint-use-prompt-regexp is a variable defined in `comint.el'.
> Its value is nil
>
> Documentation:
> If non-nil, use `comint-prompt-regexp' to recognize prompts.
> If nil, then program output and user-input are given different `field'
> properties, which Emacs commands can use to distinguish them (in
> particular, common movement commands such as `beginning-of-line'
> respect field boundaries in a natural way).
>
> You can customize this variable.
> #+end_example
>
> NIL!?  Important?
>
> It should not. The problems seem to be somewhere else... but where?  Can you
> give me a hint?[1]
>
> Best regards,
>   Seb
>
> Footnotes:
> [1] Sorry for this looooong mail. Really. But I hope its Org form will be on
> the "good" side...

-- 
Sébastien Vauban