Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)]

[Matt]

 ---- On Tue, 17 Jan 2023 14:53:39 -0500  Osher Jacob  wrote --- 
 > changing shell-command-switch to "/k" or "-k", I get a similar output:

Thanks for checking that.

 > You also mentioned the source code block is being passed through the "-c" 
 > flag as a command-line argument.I might be misunderstanding something here, 
 > but it seems like it is being passed through the stdin of the shell process 
 > when the calls process-file -> call-process are being made.

That the block source is passed through "-c" is an educated guess.  I keep 
finding myself in C land where it's less clear to me what precisely happens.  
You're correct that `call-process' acts on the block source.  It's not clear to 
me whether that's done through stdin or whether being passed through stdin 
would even help us.  Here's what I'm seeing.

Evaluating the block goes through 4 lisp commands and then into C:

1. Calls `(org-babel-eval shell-file-name (org-trim body))'

This evaluates as...

(org-babel-eval
  "cmdproxy.exe" ; command, whatever `shell-file-name' is
  (org-trim body) ; query, the block body
  )

The query parameter (the block body) gets inserted into a temp buffer on which 
`org-babel--shell-command-on-region' is called.

2. Calls `(org-babel--shell-command-on-region command error-buffer)' on a temp 
buffer containing block body.

This evaluates as...

(org-babel--shell-command-on-region
  command       ; "cmdproxy.exe"
  error-buffer  ;  #<buffer  *Org-Babel Error*>
  )

This in turn calls `process-file' where INPUT-FILE is a temp file containing 
the contents of the buffer `org-babel--shell-command-on-region' was called on 
(that is, the temp file contains the block body).

3. Calls `(process-file shell-file-name input-file (if error-file (list t 
error-file) t) nil shell-command-switch command)'

This evaluates as...

(process-file
  "cmdproxy.exe"                                               ; shell-file-name
  "/path/to/temp/containing/block/source"  ; input-file
  (t "/tmp/babel jTCHe/ob-error-yHOivA")    ; output destination (like 
call-process's DESTINATION)
  nil                                                                     ; 
display
 "-c"                                                                    ; args
  "cmdproxy.exe"                                               ; args
)

Of course, I imagine the paths would be Windows paths when run on Windows.

According to the documentation, the args are passed to the process 
(cmdproxy.exe) "verbatim without filename handling, as `call-process' does."  
The `call-process' documentation says, "remaining arguments ARGS are strings 
passed as command arguments to PROGRAM."

To me, that sounds like cmdproxy.exe gets "passed into" cmdproxy.exe.  This 
would explain the nested shell calls.  If all this were written out, I imagine 
it would look something like:

cmdproxy.exe -c cmdproxy.exe

What's not clear to me is how the INPUT-FILE is handled.  

The `process-file' documentation says "normally".  I assume it's how 
`call-process' handles INPUT-FILE, but the documentation doesn't really address 
it.

Finally, `process-file' finally calls `call-process'.

4. Calls `(apply 'call-process program (or lc infile) (if stderr-file (list 
(car buffer) stderr-file) buffer) display args)'

This evaluates as...

(apply 'call-process 
           program                                                              
    ; cmdproxy
          (or lc infile)                                                        
      ; local-copy of the INFILE,  "/path/to/temp/containing/block/source" 
          (if stderr-file (list (car buffer) stderr-file) buffer)  ; (t 
"/tmp/babel jTCHe/ob-error-yHOivA"  )
          display                                                               
      ; nil
          args)                                                                 
        ; ("-c"  "cmdproxy.exe")

How this actually works is non-trivial 
(https://git.savannah.gnu.org/cgit/emacs.git/tree/src/callproc.c#n250) and not 
something I understand at the moment.  We can see here that indeed a call like 
`cmdproxy.exe -c cmdproxy.exe' is being made.  Still, I'm not sure how the 
INPUT-FILE gets processed.  For example, is it passed in the second cmdproxy 
call?  Or, maybe it gets called first and then the second call to cmdproxy 
happens and hangs?  I don't know.

 > Any ideas on how to proceed from here?

I have two ideas.

1. Another naive work around attempt.  Again, I'm going from memory, 
documentation, and what I have previously written.  

I have in my init a command to open a terminal when working on Windows that 
looks like:

    (start-process "cmd" nil "cmd.exe" "/C" "start" "cmd.exe" "/K" "cd" dir)

This starts a cmd process with /c which terminates the prompt after the command 
that follows is run.  The command  That follows starts another cmd process with 
/k which changes the directory and leaves the terminal open.

I have another command to open QGIS that does the same thing:

    (start-process "cmd" nil "cmd.exe" "/C" "start" "\"qgis\"" "cmd.exe" "/K" 
"C:\\Program Files\\QGIS 3.22.3\\bin\\qgis.bat")

It's not clear to me why the extra call to a second cmd with /k is needed.  
Maybe it's copy-pasta or Windows being Windows.

Anyway, I mention these because I wonder if we might be able to do something 
similar.

Try changing `shell-file-name' to "cmd.exe" (ideally, the full path) and 
`shell-command-switch' to "/k" (or maybe "-k").  My hope would be that the 
final call would look something like,

cmd.exe /k cmd.exe /k <executes-input-file-as-stdin-or-whatever>

Given my analysis, I'm not sure if there's a way we could trick the call into 
being `cmd.exe /c cmd.exe /k input-file'.

2. We could write some ob-shell code to explicitly handle Windows cmd and 
powershell.  For example, call `process-file' similar to the stdin/cmdline 
branch of `org-babel-sh-evaluate'.

I'm open to this, but, like I've said, I don't have a Windows machine to work 
on right now.  I'd not be able to verify it.  I, and I'm sure others, would be 
happy to guide you if that's something you'd like to help implement.

Thoughts?