url-retrieve "Multibyte text in HTTP request" error when passing UTF-8

[Iñigo Serna]

Hi,

I'm developing a package that uses `url-retrieve` to connect to a HTTP server 
passsing a string query as a json object.

It mostly works, except when I need to pass a non-ASCII string, where it fails 
with "Multibyte text in HTTP request" error [1] raised in function 
`url-http-create-request` in url-http.el.
This happens even if I encode the query string with UTF-8.

I guess this is related to bug #23750, that in v25.1 times added a check to 
make this error happen.
If I remove that "fix" the code works without any problems.

I also tried emacs-request package [3], it works using curl backend and fails 
when using url-retrieve backend with the same error.

I tried emacs versions 26.3 (from Fedora 32), and last commits from 27.x and 
28.x branches.
The whole function is at [4].


I reread several times the bug information and I can't fully understand the 
need of that check, IMO it corresponds to the program developer, not to the API.
Anyway, is there anything I could do? Any way to pass data as a UTF-8 string?


Thanks in advance,
Iñigo Serna


[1]
Debugger entered--Lisp error: (error "Multibyte text in HTTP request: POST 
/jsonrpc.js H...")
signal(error ("Multibyte text in HTTP request: POST /jsonrpc.js H..."))
error("Multibyte text in HTTP request: %s" "POST /jsonrpc.js 
HTTP/1.1\15\nMIME-Version: 1.0\15\nConn...")
url-http-create-request()
url-http(#s(url :type "http" :user nil :password nil :host "nas" :portspec 9002 
:filename "/jsonrpc.js" :target nil :attributes nil :fullness t :silent t 
:use-cookies nil :asynchronous nil) #f(compiled-function (&rest ignored) 
#<bytecode -0x113049c86a7cc012>) (nil))
url-retrieve-internal("http://nas:9002/jsonrpc.js"; #f(compiled-function (&rest 
ignored) #<bytecode -0x113049c86a7cc012>) (nil) t t)
url-retrieve("http://nas:9002/jsonrpc.js"; #f(compiled-function (&rest ignored) 
#<bytecode -0x113049c86a7cc012>) nil t t)
url-retrieve-synchronously("http://nas:9002/jsonrpc.js"; t t 1)


[2] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=23750

[3] https://github.com/tkf/emacs-request

[4] 
(defun lms2--cmd (query &optional playerid)
  "Sync HTTP request QUERY to LMS server.  PLAYERID is optional.
QUERY is a string."
  (setq playerid (or playerid lms2--default-playerid ""))
  (let* ((json-array-type 'list)
         (url-request-method "POST")
         (url-request-extra-headers '(("Content-Type" . "application/json; 
charset=utf-8")))
         (url-request-data (json-encode-alist `[(method . "slim.request") 
(params . [,playerid ,(split-string query)])]))
         (response (with-current-buffer (url-retrieve-synchronously (concat 
lms2-url "/jsonrpc.js") t t 1)
                     (goto-char (point-min))
                     (re-search-forward "^$")
                     (delete-region (point) (point-min))
                     (decode-coding-string (buffer-string) 'utf-8))))
    (unless (string= "" response)
      (alist-get 'result (json-read-from-string response)))))

(lms2--cmd (encode-coding-string "artists 0 10 search:Björk" 'utf-8))