The benchmark of Artanis: guile server, Fibers, and Ragnarok

[Nala Ginrut]

This is not a serious benchmark, since Artanis is still not  in the
stage for optimizing.

The test case is from the first test of Techempower, the json serialization:

https://www.techempower.com/benchmarks/#section=data-r13&hw=ph&test=json

According to Techempower, the response should look like:
--------------------------------------------
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 28
Server: Example
Date: Wed, 17 Apr 2013 12:00:00 GMT

{"message":"Hello, World!"}
---------------------------------------------


Linux system config:
-----------------/etc/sysctl.conf----------------
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 5000
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_timestamps = 0
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
----------------------------------------------------------


The Artanis code for test:
-------------code--------------
(get "/json" #:mime 'json
  (lambda (rc)
     (:mime rc "{\"message\":\"Hello, World!\"}")))
--------------end---------------

The benchmark tool is wrk:
-----------------------------------
sudo apt-get install wrk
-----------------------------------


The platform:
--------------------
Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
16G RAM
Debian9
GNU Artanis-0.2.5
Guile-2.2.3
Fibers  (commit b86405a2)
--------------------

The command:
-----------
wrk -c 100 -t 4 -d 10 http://localhost:3000/json
-----------
It means 100 connections keep open, 4 threads, 10 seconds.


======
1. Guile builtin server

----------run--------------
art work -s guile
-----------------------------

-c 100
--------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    81.66ms  158.75ms   1.98s    96.69%
    Req/Sec   419.24     66.16   777.00     79.75%
  16704 requests in 10.01s, 2.88MB read
  Socket errors: connect 0, read 0, write 0, timeout 10
Requests/sec:   1669.45
Transfer/sec:    295.09KB
-----------------

-c 1000
----------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   203.02ms  121.22ms   1.90s    59.75%
    Req/Sec   428.52    115.43   780.00     70.25%
  17076 requests in 10.06s, 2.95MB read
  Socket errors: connect 0, read 0, write 0, timeout 39
Requests/sec:   1697.54
Transfer/sec:    300.05KB
-----------------


==========
Fibers

----------run---------
art work -s fibers
------------------------

-c 100
----------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    61.13ms    3.19ms  70.30ms   88.85%
    Req/Sec   409.62     79.96   505.00     55.25%
  16321 requests in 10.01s, 2.82MB read
Requests/sec:   1629.75
Transfer/sec:    288.07KB
----------------

-c 1000
--------------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   613.76ms   79.96ms 666.43ms   95.09%
    Req/Sec   395.84    121.35   707.00     68.75%
  15772 requests in 10.06s, 2.72MB read
Requests/sec:   1567.32
Transfer/sec:    277.04KB
--------------------


===============
Ragnarok
(server.multi requires Linux-3.9+)

-----------run----------
art work -s ragnarok
--------------------------

-c 100
----------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    95.32ms  106.54ms 546.36ms   83.09%
    Req/Sec   391.62    338.22     1.58k    76.96%
  15444 requests in 10.01s, 2.67MB read
Requests/sec:   1542.98
Transfer/sec:    272.73KB
----------------

-c 100 (enable multi-server, 4 instances)
----------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    26.26ms   24.61ms 149.54ms   54.36%
    Req/Sec     1.12k   166.39     1.82k    66.50%
  44749 requests in 10.01s, 7.72MB read
Requests/sec:   4468.86
Transfer/sec:    789.91KB
----------------

-c 1000
---------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   284.85ms  515.70ms   1.97s    82.14%
    Req/Sec   411.60    363.25     1.68k    74.80%
  15838 requests in 10.06s, 2.73MB read
  Socket errors: connect 0, read 0, write 0, timeout 1556
Requests/sec:   1574.00
Transfer/sec:    278.22KB
---------------

-c 1000 (enable multi-server, 4 instances)
----------------
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   209.78ms  154.24ms 559.29ms   53.19%
    Req/Sec     1.18k   365.76     2.56k    70.00%
  47131 requests in 10.06s, 8.14MB read
Requests/sec:   4684.08
Transfer/sec:    827.95KB
----------------



Seems there're lot of things need to be optimized. But in the
anticipated future, I'll only focus on stability and more features.

And please don't forget that the Guile compiler/VM has large room to
optimize, we can hope for the future.

If anyone wants to write a server-core with libuv or libev, feel free
to do it, I'll provide support if any necessary.

Happy hacking!