bug#20744: shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly

[Da Zhang]

From: Da Zhang <zhangda82@gmail.com>
To: bug-gnu-emacs@gnu.org
Subject: 24.5; shell-quote-argument in Emacs 24.5 and Emacs 25 on Windows cannot figure out path correctly
--text follows this line--


Dear Emacs developers/maintainers,

I am writing this email to report an issue that has been bothering me
for a while.

I use python mode (python.el) for some python programming. When I invoke
the command python-shell-send-buffer, my Emacs always reports "Searching
for program: no such file or directory,
c\:/Anaconda3/Scripts/ipython.exe".

I traced down the cause of this problem, and found the following lines
in python.el:
(defun python-shell-parse-command ()    ;FIXME: why name it "parse"?
  "Calculate the string used to execute the inferior Python process."
  ;; FIXME: process-environment doesn't seem to be used anywhere within
  ;; this let.
  (let ((process-environment (python-shell-calculate-process-environment))
        (exec-path (python-shell-calculate-exec-path)))
    (format "%s %s"
            ;; FIXME: Why executable-find?
            (shell-quote-argument (executable-find python-shell-interpreter))
            python-shell-interpreter-args)))

The function call
(shell-quote-argument (executable-find python-shell-interpreter))
changed the path of my python interpreter from
"c:/Anaconda3/Scripts/ipython.exe"
to
"c\\:/Anaconda3/Scripts/ipython.exe".

I worked around this problem by replacing the function call
(shell-quote-argument (executable-find python-shell-interpreter))
with
(executable-find python-shell-interpreter) ;; edited by DZ on
2015/06/05: (shell-quote-argument) caused the problem of
"c\\:/Anaconda3/Scripts/ipython.exe"

To further analyze the problem, I found the function
shell-quote-argument defined in subr.el does not deal with the case when
bash from cygwin is used.

(defun shell-quote-argument (argument)
  "Quote ARGUMENT for passing as argument to an inferior shell."
  (cond
   ((and (eq system-type 'windows-nt) (w32-shell-dos-semantics))
    ;; First, quote argument so that CommandLineToArgvW will
    ;; understand it.  See
    ;; http://msdn.microsoft.com/en-us/library/17w5ykft%28v=vs.85%29.aspx
    ;; After we perform that level of quoting, escape shell
    ;; metacharacters so that cmd won't mangle our argument.  If the
    ;; argument contains no double quote characters, we can just
    ;; surround it with double quotes.  Otherwise, we need to prefix
    ;; each shell metacharacter with a caret.

    (setq argument
          ;; escape backslashes at end of string
          (replace-regexp-in-string
           "\\(\\\\*\\)$"
           "\\1\\1"
           ;; escape backslashes and quotes in string body
           (replace-regexp-in-string
            "\\(\\\\*\\)\""
            "\\1\\1\\\\\""
            argument)))

    (if (string-match "[%!\"]" argument)
        (concat
         "^\""
         (replace-regexp-in-string
          "\\([%!()\"<>&|^]\\)"
          "^\\1"
          argument)
         "^\"")
      (concat "\"" argument "\"")))
   (t
    (if (equal argument "")
        "''"
      ;; Quote everything except POSIX filename characters.
      ;; This should be safe enough even for really weird shells.
      (replace-regexp-in-string
       "\n" "'\n'"
       (replace-regexp-in-string "[^-0-9a-zA-Z_./\n]" "\\\\\\&" argument))))
   ))

I added the following lines in the cond statement, to solve the problem.
   ((and (eq system-type 'windows-nt) (if (string-match ".*cygwin.*" (w32-shell-name)) t nil)) ;; DZ's edit on 6/5/2015
    (if (equal argument "")
        "''"
      ;; Quote everything except POSIX filename characters.
      ;; This should be safe enough even for really weird shells.
      (replace-regexp-in-string
       "\n" "'\n'"
       (replace-regexp-in-string "[^-0-9a-zA-Z_./\n:]" "\\\\\\&" argument))))
   ((eq system-type 'ms-dos)
    ;; Quote using double quotes, but escape any existing quotes in
    ;; the argument with backslashes.
    (let ((result "")
          (start 0)
          end)
      (if (or (null (string-match "[^\"]" argument))
              (< (match-end 0) (length argument)))
          (while (string-match "[\"]" argument start)
            (setq end (match-beginning 0)
                  result (concat result (substring argument start end)
                                 "\\" (substring argument end (1+ end)))
                  start (1+ end))))
      (concat "\"" result (substring argument start) "\"")))

However, the edits does not change the behavior of subr.el.

I worked with Eli Zaretskii on similar problem before, and here I report a more thorough analysis in the hope some experts more capable than me could help to solve this problem.

Thanks.

Da Zhang



In GNU Emacs 24.5.1 (x86_64-w64-mingw32)
 of 2015-05-16 on KAEL
Windowing system distributor `Microsoft Corp.', version 6.3.9600
Configured using:
 `configure --prefix=/z/emacs --host=x86_64-w64-mingw32
 --target=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --with-wide-int
 --with-jpeg --with-xpm --with-png --with-tiff --with-rsvg --with-xml2
 --with-gnutls --with-sound=yes --with-file-notification=yes
 --without-dbus --without-imagemagick 'CFLAGS=-O3 -fomit-frame-pointer
 -g0 -pipe' 'LDFLAGS=-static-libgcc -static-libstdc++ -static -s
 -Wl,-s''

Important settings:
  value of $LANG: ENU
  locale-coding-system: utf-8

Major mode: Emacs-Lisp

Minor modes in effect:
  guide-key-mode: t
  recentf-mode: t
  winner-mode: t
  cua-mode: t
  global-company-mode: t
  company-mode: t
  rainbow-delimiters-mode: t
  sml-modeline-mode: t
  global-undo-tree-mode: t
  undo-tree-mode: t
  anything-dired-mode: Enable anything completion in Dired functions.
Bindings affected are C, R, S, H.
This is deprecated for Emacs24+ users, use `ac-mode' instead.
  ido-vertical-mode: t
  ido-everywhere: t
  tabbar-mwheel-mode: t
  tabbar-mode: t
  shell-dirtrack-mode: t
  desktop-save-mode: t
  TeX-PDF-mode: t
  global-auto-revert-mode: t
  display-time-mode: t
  auto-image-file-mode: t
  show-paren-mode: t
  which-function-mode: t
  delete-selection-mode: t
  global-hl-line-mode: t
  diff-auto-refine-mode: t
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t
  abbrev-mode: t

Recent messages:
Starting new Ispell process aspell with default dictionary...
Fontifying README.W32... (regexps...........................................)
Starting new Ispell process aspell with default dictionary...
Wrote c:/home/zhangda/.emacs.d/.emacs.desktop.lock
Desktop: 1 frame, 5 buffers restored.
(Shell command succeeded with no output)
Desktop: 1 frame, 5 buffers restored.
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark saved where search started [2 times]
Mark activated

Load-path shadows:
c:/home/zhangda/.emacs.d/elpa/anything-20130606.946/anything-complete hides c:/home/zhangda/.emacs.d/elpa/anything-complete-1.86/anything-complete
c:/home/zhangda/.emacs.d/elpa/anything-20130606.946/anything-match-plugin hides c:/home/zhangda/.emacs.d/elpa/anything-match-plugin-1.27/anything-match-plugin

Features:
(shadow sort mail-extr warnings emacsbug sendmail misearch multi-isearch
swiper ivy darkroom eim eim-extra cdlatex texmathp flyspell ispell
org-rmail org-mhe org-irc org-info org-gnus org-docview doc-view
jka-compr image-mode org-bibtex bibtex org-bbdb org-w3m indent-guide
python-cell outline-magic auto-complete popup highlight-indentation
sphinx-doc guide-key popwin company-files company-oddmuse
company-keywords company-etags company-gtags company-dabbrev-code
company-dabbrev company-capf company-cmake company-xcode company-clang
company-semantic company-eclim company-template company-css company-nxml
company-bbdb vc-hg recentf tree-widget helm-files helm-buffers
helm-elscreen helm-tags helm-bookmark helm-adaptive helm-info bookmark
pp helm-locate helm-help helm-match-plugin helm-grep helm-regexp
helm-plugin helm-external helm-net helm-utils helm helm-source f s
ucs-normalize ace-jump-zap ace-jump-mode winner smartparens-config
smartparens-python smartparens-latex smartparens dash cua-base
visual-regexp eldoc company elpy pyvenv elpy-refactor files-x etags
flymake-cursor flymake-python-pyflakes flymake-easy
python-el-fgallina-expansions python json iedit iedit-lib smex
rainbow-delimiters re-builder matlab-load rainbow-mode google-c-style
fbib sml-modeline framemove windmove weblogger xml-rpc timezone url-http
tls url-auth url-gw expand-region text-mode-expansions
cc-mode-expansions the-org-mode-expansions latex-mode-expansions
er-basic-expansions expand-region-core expand-region-custom undo-tree
diff anything-config browse-url rx anything-match-plugin xml url
url-proxy url-privacy url-expand url-methods url-history url-cookie
url-domsuf url-util mailcap tramp tramp-compat tramp-loaddefs trampver
ffap url-parse auth-source eieio eieio-core gnus-util password-cache
url-vars anything bm find-dired help-mode dired+ image-dired dired-x
dired-aux header2 color-theme-dawn-night color-theme ido-vertical-mode
ido browse-kill-ring tabbar ibuf-macs ibuffer server org-bullets
ox-beamer ox-latex ox-icalendar ox-html ox-ascii ox-publish ox
org-element ob-makefile ob-sh shell ob-R ob-python ob-perl ob-org
ob-matlab ob-octave ob-latex ob-gnuplot ob-plantuml ob-ditaa ob-css
ob-calc calc-store calc-trail calc-ext calc calc-loaddefs calc-macs ob-C
cc-mode cc-fonts cc-guess cc-menus cc-cmds cc-styles cc-align cc-engine
cc-vars cc-defs org org-macro org-footnote org-pcomplete pcomplete
org-list org-faces org-entities noutline outline org-version
ob-emacs-lisp ob ob-tangle ob-ref ob-lob ob-table ob-exp org-src ob-keys
ob-comint ob-core ob-eval org-compat org-macs org-loaddefs find-func
cal-menu calendar cal-loaddefs muse-wiki muse-colors muse-docbook
muse-texinfo texnfo-upd texinfo muse-latex muse-html muse-xml-common
cus-edit cus-start cus-load muse-publish muse-project muse-protocols
muse-regexps wid-edit derived muse muse-nested-tags muse-mode flymake
tex-mik preview prv-emacs reporter desktop frameset latex tex-style
tex-buf tex crm cl-macs autorevert filenotify time image-file avoid
paren which-func imenu delsel col-highlight vline highlight-symbol
thingatpt highlight-parentheses hl-line byte-opt bytecomp byte-compile
cl-extra cconv hlinum linum edmacro kmacro w32-browser htmlize-view
htmlize cl gv cl-loaddefs cl-lib ahg vc-annotate vc vc-dispatcher dired
grep compile ewoc log-edit message format-spec rfc822 mml mml-sec
mm-decode mm-bodies mm-encode mail-parse rfc2231 rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mailabbrev mail-utils gmm-utils mailheader
pcvs-util add-log diff-mode easy-mmode cygwin-mount ange-ftp comint
ansi-color ring anything-autoloads anything-complete-autoloads
anything-match-plugin-autoloads color-file-completion-autoloads
color-theme-autoloads tex-site csv-mode-autoloads etags-table-autoloads
flymake-shell-autoloads gnuplot-mode-autoloads hide-lines-autoloads
htmlize-autoloads http-post-simple-autoloads less-autoloads
muse-autoloads py-import-check-autoloads pydoc-info advice help-fns
info-look rainbow-mode-autoloads info easymenu sml-mode-autoloads
vline-autoloads weblogger-autoloads package epg-config time-date tooltip
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel dos-w32
ls-lisp w32-common-fns disp-table w32-win w32-vars tool-bar dnd fontset
image regexp-opt fringe tabulated-list newcomment lisp-mode prog-mode
register page menu-bar rfn-eshadow timer select scroll-bar mouse
jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
w32notify w32 multi-tty emacs)

Memory information:
((conses 16 670478 66958)
 (symbols 56 63650 0)
 (miscs 48 489 281)
 (strings 32 157872 41458)
 (string-bytes 1 4522676)
 (vectors 16 75479)
 (vector-slots 8 1116100 22330)
 (floats 8 400 929)
 (intervals 56 3470 0)
 (buffers 960 18))