[lwip-users] TCP tuning to increase throughput

[Mandeep Sandhu]

Hi All,

I have small HTTP proxy server implemented using the lwIP sequential API.

When a client connects to my webserver, the webserver proxies the
request to another server sitting somewhere on the internet. The
response of this server is then written back as-is to the client
connection. All this while the client is waiting on the open socket
connection.

Roughly this is what I do:

netconn_write( proxy_conn, <data to be proxied>, <length>, NETCONN_COPY);

   while ((proxy_resp = netconn_recv(proxy_conn)) != NULL ) {
       do {
           netbuf_data(proxy_resp, &data, &len);
           //Write response back to the client
           err = netconn_write(client_conn, data, len, NETCONN_COPY);

           if (err != ERR_OK) {
               goto error; // some error handler
           }
       } while (netbuf_next(proxy_resp) >= 0);
       netbuf_delete(proxy_resp);
   }

(this code is based on the TCP echo example)

When I put debug's in the inner loop, I saw that the other end was
sending data in chunks of 128 bytes. Even after I increase the MSS to
256 bytes. I was using wget on the client side, and it was reporting a
speed of ~150 KB/s, where as a direct get of the same data from the
server directly would be ~ 4 MB/s.

Also, I saw that if the data was large (as in a large page ~200 KB), I
was getting mem alloc error:

memp_malloc: out of memory in pool TCP_PCB

I've increased the pool size, but that seems to only mitigate the
problem temporarily. Now the mem alloc error happens at slightly
larger values (~400KB).

I'm not very conversant with TCP, so any tips on how to increase
throughput to the max and avoid the mem issue? The PBUF buffer size is
set to 182 bytes.

Thanks,
-mandeep