Re: Trouble parsing a response

[Roel Janssen]

On 13-12-18 23:29, address@hidden wrote:
> On 2018-12-13 23:03, Roel Janssen wrote:
> > On 13-12-18 17:06, address@hidden wrote:
> > > On 2018-12-13 16:01, address@hidden wrote:
> > > snip
> > >
> > > > I tried with the file attached but got this because the driver does not
> > > > support URIs but only host, port, type, token:
> > >
> > > Ah, I saw now that you already implemented URI on master :)
> > > https://github.com/roelj/guile-sparql/blob/master/sparql/driver.scm
> > >
> > > When I try calling this
> > > ;; Example query to wikidata listing cats
> > >                                          (sparql-query
> > >    "SELECT ?item
> > >                                          WHERE
> > >                                          {
> > >                                          ?item wdt:P31 wd:Q146.
> > >                                          }
> > >                                          LIMIT 10
> > >                                          "
> > >    #:uri "https://query.wikidata.org/sparql";
> > >    ;; #:port 80
> > >                                            #:type 
> > > "application/sparql-results+json"
> > >    ;;  #:token "..."
> > >                                            #:store-backend 'blazegraph
> > >    )
> > >
> > > I get this fine result:
> > > #<<response> version: (1 . 1) code: 200 reason-phrase: "OK" headers:
> > > ((date . #<date nanosecond: 0 second: 12 minute: 32 hour: 15 day: 13
> > > month: 12 year: 2018 zone-offset: 0>) (content-type
> > > application/sparql-results+json (charset . "utf-8")) (transfer-encoding
> > > (chunked)) (connection close) (server . "nginx/1.13.6") (x-served-by .
> > > "wdqs1005") (access-control-allow-origin . "*") (cache-control public
> > > (max-age . 300)) (vary accept accept-encoding) (x-varnish . "644531744,
> > > 572094009, 417977651") (via "1.1 varnish (Varnish/5.1)" "1.1 varnish
> > > (Varnish/5.1)" "1.1 varnish (Varnish/5.1)") (accept-ranges bytes) (age .
> > > 0) (x-cache . "cp1079 pass, cp3030 pass, cp3030 pass") (x-cache-status .
> > > "pass") (server-timing . "cache;desc=\"pass\"")
> > > (strict-transport-security . "max-age=106384710; includeSubDomains;
> > > preload") (set-cookie .
> > > "WMF-Last-Access=13-Dec-2018;Path=/;HttpOnly;secure;Expires=Mon, 14 Jan
> > > 2019 12:00:00 GMT") (set-cookie .
> > > "WMF-Last-Access-Global=13-Dec-2018;Path=/;Domain=.wikidata.org;HttpOnly;secure;Expires=Mon,
> > > 14 Jan 2019 12:00:00 GMT") (x-analytics . "https=1;nocookies=1")
> > > (x-client-ip . "83.185.90.53")) port: #<input-output: file 8c37e70>>
> > >
> > > My problem now is that I don't know how to separate the header from the
> > > port-file.
> > >
> > > Ah, reading here
> > > https://www.gnu.org/software/guile/manual/html_node/Responses.html#Responses
> > > I found (response-port).
> >
> > Here's what I did in SPARQLing-genomics:
> > https://github.com/UMCUGenetics/sparqling-genomics/blob/master/web/www/pages/query-response.scm#L86
> >
> > So basically:
> >
> > (use-modules
> >    (ice-9 receive)
> >    (ice-9 rdelim)
> >    (web response))
> >
> > (receive (header port)
> >    ;; Note: "text/csv" is the only format that is consistent for
> > multiple SPARQL back-ends (Virtuoso, BlazeGraph, ...)
> >    (sparql-query ... #:type "text/csv")
> >    (if (= (response-code header) 200) ; This means the query went OK.
> >      (call-some-function port)
> >      #f)) ; Deal with errors at the #f.
> >
> > (define (call-some-function port)
> >    (let ((line (read-line port)))
> >      (if (eof-object? line)
> >        #t
> >        (begin
> >          (format #t "Line: ~a~%" line)
> >          ;; Tail-recurse until we have processed each line.
> >          (call-some-function port)))))
>
> This would be a really nice addition to the guile manual. Can I submit
> it as a patch crediting you?

Sure!

> > The SPARQLing-genomics code deals with more error codes, and processes
> > the lines in a more useful way.
> >
> > > Unfortunately this only took me one step further as I run into this
> > > instead when trying to parse the port with (json->scm):
> > >
> > > Backtrace:
> > >              7 (apply-smob/1 #<catch-closure 9769550>)
> > > In ice-9/boot-9.scm:
> > >       705:2  6 (call-with-prompt _ _ #<procedure default-prompt-handle…>)
> > > In ice-9/eval.scm:
> > >       619:8  5 (_ #(#(#<directory (guile-user) 9759910>)))
> > > In ice-9/boot-9.scm:
> > >      2312:4  4 (save-module-excursion _)
> > >     3831:12  3 (_)
> > > In sdb-test.scm:
> > >        24:1  2 (_)
> > > In json/parser.scm:
> > >      311:18  1 (json-read-number _)
> > >      148:28  0 (read-number _)
> > >
> > > json/parser.scm:148:28: In procedure read-number:
> > > Throw to key `json-invalid' with args `(#<json-parser port:
> > > #<input-output: string 98dea80>>)'.
> > >
> > > Maybe this is a bug in (json)?
> >
> > It looks like the JSON response is not (only) JSON, or simply invalid.
> > Maybe the "text/xml" or "text/csv" content-type will work better for
> > you.  I noticed that each back-end provides their own structure for
> > XML and JSON, so I used the somewhat quirky CSV format as a
> > work-for-all response type.
>
> Ok, good to know. Is this documented in your guile-sparql docs?

No.  I think what SPARQL endpoints actually return is beyond the scope 
of guile-sparql, and may change in the future anyway.

> > I hope this helps.
>
> Thanks a lot for taking the time to write this. :)
> I already met receive in the npm importer but I did not really learn to
> use it yet. So whenever I see multiple objects returned I can part them
> with receive! Nice.

Exactly.  I'm glad my snippet helped.

Kind regards,
Roel Janssen