Re: Emacs HTTP libraries [was: Re: How to contribute new package to GNU ELPA?]

[Philip K.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Philip K." <philipk@posteo.net>
>> Cc: arthur.miller@live.com, adam@alphapapa.net, rms@gnu.org,
>>      emacs-devel@gnu.org
>> Date: Tue, 22 Dec 2020 00:51:08 +0100
>> 
>> >> > Before this is taken as an indication that using one of these
>> >> > libraries will automagically make Emacs as fast as the applications
>> >> > like curl, we should carefully profile url.el and find out which
>> >> > part(s) of it cause the slowness.  Because it could well be that what
>> >> > makes url.el slow will also make Emacs using libcurl slow.
>> >> 
>> >> Maybe this library can be of help while testing/profiling
>> >
>> > We have a built-in profiler in Emacs.
>> 
>> In that case, gnutls-negotiate seems to be the most suspicious function,
>> using over 50%-60% of the CPU time, at least on my machine. This also
>> makes sense, as TLS sites seem to take longer to load than regular,
>> non-encrypted ones.
>
> Please show the code you profiled and the fully expanded profile.

I sadly coudln't reproduce it, but this time the critical section looked
something like this:

        - url-retrieve-synchronously                               212  52%
         - url-retrieve                                            149  36%
          - url-retrieve-internal                                  149  36%
           - url-http                                              136  33%
            - url-http-find-free-connection                        135  33%
             - url-open-stream                                     135  33%
                open-network-stream                                134  33%

when evaluating 

        (url-retrieve-synchronously "http://textboard.org/sexp/prog/index";)

in the *scratch* buffer. I used emacs -Q (GNU Emacs 27.1 (build 1,
x86_64-redhat-linux-gnu, GTK+ Version 3.24.22, cairo version 1.16.0) of
2020-08-21), but I don't know why that should make any
difference. I repeated the same test on the master branch and the
results were basically the same (±5%).

Either way, this simple request took over 2.5 minutes, whereas curl
requires a quarter of a second. Note that this is even unencrypted, so
this is not even taking the encryption overhead into account.

(FYI: I took textboard.org as an example, because I wrote a client for
      this site in Elisp, that often takes a while to load, so I could
      count on it for this test. An interesting observation is that the
      first request might take a while, but sometimes the connection
      time suddenly drops to basically nothing. I have yet to figure
      out what the reason is. :/)

-- 
        Philip K.