Re: [lwip-users] TCP speed drops to a crawl.

[Kieran Mansley]

On Thu, 11 Aug 2005, Michael Williamson wrote:

> Hi,
>
> It appears that altering the MEM_SIZE (not TCP_SEND_SIZE) has a linear impact 
> on when this issue occurs.  When it's set to 1600, we drop out at 3100 
> packets, when it's set to 512000 we drop out at ~100000 packets.

There are various resources that are in in short supply.  If you run out 
of one of these, such as packet buffers or pcbs, you'll not be able to 
continue until you're able to free some more.  Receiving an ACK is likely 
to free some resources, which is why it then sends some more.

>  With 
> TCP_NODELAY set, I would expect that the rate of transmission not be tied 
> one-to-one with the client ack rate, no?  That's the behaviour when things 
> slow down.

That's exactly what I'd expect, although maybe I'm misunderstanding your 
problem.  With TCP in a steady state (i.e. after the initial slow start, 
and so on) you can't send data faster than it is acknowledged.  If you 
receive an acknowledgement that acknowledges "X" bytes, you can send "X" 
bytes more.  If you send a substantial amount of data (the accepted 
definition of "substantial" varies, but most agree on 2 max segment sized 
packets) and the other end isn't acknowledging it immediately, the other 
end is at fault.  It sounds from your description (200ms delay between 
ACKs) that the other end is using a delayed ACK strategy, which in turn 
suggests that you're not sending very much data (i.e. < 2 max 
segment sized packets) between each ACK.

> A couple of other test observations:
>
> When large messages are used (i.e., messages that require breaking into 
> multiple packets, say 10Kbytes), this problem does not present itself.  Seems 
> to be related to smaller packet sizes.
>
> If you disconnect and then reconnect the client, the rates go back up and 
> then drop again.  It's very reproducable.
>
> I'm beginning to wonder if there is a problem cleaning up buffers tied to 
> large groups of ack'd packets?  Is there a limit to the queue size of 
> unacknowledged buffers that could be getting overrun?

It's been a while since I looked at the code, but my guess would be you're 
running out of packet buffers or something like that - the fact that when 
you use large packets rather than small ones the problem goes away backs 
this up.

It's going to be hard to work out more from here.  Perhaps a copy of an 
ethereal trace would help explain what's going wrong.

Kieran