So just an update on what I've managed to do to get things working -
I think I was confused about a few lwip things and distracted by xilinx overhead other times.
The first thing that confused me was that when you successfully tcp_close() a connection, the tcp_pcb pointer that you made will still be a valid pointer to a tcp_pcb struct - the memory is not actually cleared. In addition, the state the struct returns to is identical to the state of a brand new tcp_pcb, "CLOSED". This caused me to believe that I could tcp_connect() again on the same tcp_pcb. As you've said, this is not actually ok. After tcp_close returns, lwip no longer knows about your tcp_pcb struct, and although the memory isn't cleared, lwip has designated that memory space as free so the next time you create a connection with tcp_new() it may place it there and overwrite your old tcp_pcb.
So my four situations:
1) I ctrl-C the listener on the other machine.
This will send some sort of packet to your lwip machine telling you the connection is closing. Lwip will move tcp_pcb into the CLOSE_WAIT state, and will call the recv_callback with a NULL argument. In the recv_callback you should then check that this packet is coming from your currently opened connection. If it isn't, it must be from an old connection that you already attempted to close, so you can ignore it. If it is, then you must call tcp_close() yourself. This then completes some more acking with the other machine and eventually places the tcp_pcb into the CLOSED state, and then calls the err_callback with ERR_ABRT. At this point, the tcp_pcb is "freed" and should not be touched again. A new tcp_pcb should be tcp_new()'d and then you can tcp_connect() again.
2) I try to connect from lwip but no cable is plugged in.
Lwip will try and send a SYN packet and wait for an ack. Until it gets the ack, a counter is being incremented until it times out. When it times out, it calls the err_callback with ERR_ABRT. It is not clear to me if the tcp_pcb is CLOSED at this point, but it looks like the memory is freed anyway. Should I have to call tcp_close() here? Once the memory is freed in lwip you can again tcp_new() and tcp_connect().
3) I try to connect from lwip and the cable is plugged in, but nothing is listening.
The other machine should immediately send Lwip back a rst packet. Lwip will then call the err_callback with ERR_RST. I noted though that at this point when the error callback is called, the old tcp_pcb is not yet closed nor freed. I was worried here that creating a new connection before the old one was closed would mess something up since the old connection is still in lwips active pcbs lists. Currently I just wait until the state of the old connection is closed, knowing that lwip will immediately close and free it after calling the callback. I can then tcp_new() and tcp_connect().
4) I connect fine, but then unplug the cable. My manually sent ping never gets a response.
If you call tcp_close() but the other machine is not connected to respond, the close request wont get acked and the tcp_pcb will sit in FIN_WAIT_1 forever. I guess it is ok to leave this connection hanging and start a new one. You wont get any callbacks so you'd have to call tcp_new() immediately after calling tcp_close(). When you eventually reconnect, the old connection will close and call the recv_callback with a NULL argument, but as described above you will know it's not from the current connection so you will ignore it anway. On the other hand, if you call tcp_abort(), lwip immediately gets rid of it and sets the connection to CLOSED, and then err_callback is called with ERR_ABRT. At this point you can tcp_new() and tcp_connect() again.
The other weird detail for me was that every time I closed the connection I had to re call the xilinx function platform_enable_interrupts() for the lwip timers to work. For some reason xilinx automatically turns off the interrupts whenever the connection was closed.
Hopefully this is somewhat accurate. At the least my code is stable for now.
Thanks,
Richie