Re: Some help needed to use curl lib to download binary file

[Vivien Kraus]

Hello,

I see in the paste:

> ;; function taken on
> https://gist.github.com/amirouche/138a27bdbef5a672a0135f90ca26ec41
> ;; then adapted to use cookie jar
> (define-public (http-get url cookie-exist)
>   ;; Create a Curl handle
>   (let ((handle (curl-easy-init)))
>     ;; Set the URL from which to get the data
>     (curl-easy-setopt handle 'url url)
>     (if cookie-exist
>         (curl-easy-setopt handle 'cookie "cookie.txt")
>         (curl-easy-setopt handle 'cookiejar "cookie.txt"))
> 
>     ;; Request that the HTTP headers be included in the response
>     (curl-easy-setopt handle 'header #t)
>     ;; Get the result as a Latin-1 string
>     (let* ((response-string (curl-easy-perform handle))
>            ;; Create a string port from the response
>            (response-port (open-input-string response-string))
>            ;; Have the (web response) module to parse the response
>            (response (read-response response-port))
>            (body (utf8->string (read-response-body response))))
>       (close response-port)
>       ;; Have the (web response) module extract the body from the
>       ;; response
>       (values response body))))

So here the call expects the response to be UTF-8 text. If it is a
binary file that you are downloading, the function will raise an
exception. Guile has that python3 feeling where you are supposed to
know in advance whether what you are using is text or binary, which is
hurting you here. 

However, you can avoid the problem by either having bytevectors
everywhere, so removing the call to utf8->string and bind the "body"
variable directly to (read-response-body response), or pretend that you
know better than guile and pretend that it is latin-1-encoded, so you
lose no information and guile won’t complain. In that case, load (ice-9
iconv) and replace (utf8->string …) with (bytevector->string … "ISO-
8859-1").

If you go the first route, you get a bytevector back. If you go the
second one, you get a string, but you must remember that it contains
raw bytes and not text (unless the response body was indeed text).

There are some cases when you might want to have such strings-that-
contain-binary-or-text, such as if you want to use the strings API on
them and the bytevector API does not provide what you want, or you want
to interface with NUL-terminated strings. In other cases you might
prefer bytevectors. Anyway, there is no encoding cost to convert
between strings-that-contain-binary-or-toxt and bytevectors, it is as
simple as a copy.

> ;; write content into file
> (call-with-output-file “download.zip” (lambda (current-output-port)
>                                         (get-file get-file-link)
>                                         (put-bytevector (current-
> output-port) body)))

call-with-output-file calls its function argument with a port. Plus,
you just ignore the result of get-file, so instead you should do
something with it.

If you chose to have get-file return a bytevector, you can do:

(call-with-output-file "download.zip"
  (lambda (port)
    (put-bytevector port (get-file get-file-link)))
  #:binary #t)

If get-file returns a string, you can do:

(call-with-output-file "download.zip"
  (lambda (port)
    (put-string port (get-file get-file-link)))
  #:encoding "ISO-8859-1")

Best regards,

Vivien