Re: [lwip-users] recv ok.. send is super slow

[Kieran Mansley]

On Tue, 2008-05-13 at 09:07 -0400, Rishi Khan wrote:

> The original send problem turned out that it would send one packet,
> wait for the ack, then send the next. This was because the
> TCP_SND_BUF was only 2048 in the unixsim contrib folder, which I used
> as a basis.

The values used in the examples are often very conservative defaults,
and not tuned for performance.  lwIP is in general targeted at low
memory systems rather than high performance ones, which is why the
defaults lean in that direction.  Some time ago someone posted a list of
sensible values to use for performance.  These may have made it onto the
lwIP wiki, but I'm not sure.  Searching the mailing list archives might
help if not.

> My test program linux-to-linux gets 117.8MB/s (940Gbps). It gets
> 113.2MB/s with TCP window scaling off, but this is off topic.
> Linux_send-lwip_recv gets 923.3MB/s (738Gbps). Linux_send-lwip_recv
> gets 92.3MB/s (738Gbps). lwip_both gets 92.3MB/s (738Gbps).

I think you've confused Gbps and Mbps, and got a decimal place or two in
the wrong place in those figures, but I think I can make sense of them.
It looks like you're getting pretty good performance from lwIP.  I
wouldn't expect it to beat the native Linux stack.

> It seems that on the recv side, LWIP sends multiple requests for TCP  
> window_size updates.

I think we have an outstanding bug about window updates.  Might be worth
checking to see if that matches your observed behaviour.

> The window size drops significantly and  
> fluctuates between 30K and 60K. Also, there seems to be a lot of PSH  
> flags when lwip sends. The packets are truncated, but the data  
> payload is just '0x7a7a7a7 ....'

I'll try and find some time to take a look at your packet captures.

> Finally, somehow linux can send packets bigger than the 1500 MTU. How  
> is this possible? Is it fragmenting the packets? The IP flags say  
> 'don't fragment'.

Most likely Linux is doing segmentation offload to your NIC.  i.e.
passing a larger than MTU socket buffer to the NIC, which then fragments
it into MTU sized chunks.  This is a pretty standard feature in modern
NICs to reduce the overheads on the network stack. 

Thanks

Kieran