lwip-users
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[lwip-users] httpc_get_file doesn't seem to download entire file?


From: Bas Prins
Subject: [lwip-users] httpc_get_file doesn't seem to download entire file?
Date: Wed, 9 Jun 2021 12:42:44 +0200

Hi

I am having a hard time understanding the http_client.h interface. 

My setup:
- freertos latest version
- lwip 2.1.2 (over pppos)
- the only tasks running are lwip related (tcpip thread (task prio=2 + rx thread (task prio=2).
- both are over dimensioned for stack and heap (because I'm still developing, I'll reduce to decent amounts when I need to)
- uart2 rx/tx are connected to LTE modem. Both rx and tx is used in interrupt mode.
- uart3 tx is used for logging to PC (where some simple C# app reads all data and logs to file)

I am connecting to a web-server where I want to download a "512.KB.zip" (zip contains \0 characters, I took it from some download speed test server).

I experimented with lwftp, but it seems to leak memory the way I use it. So I thought, why not just download files using http get and use the lwip provided http_client.

The following code is used to initiate the download :

    LogLineToUart("connecting to server...\n");

    ip_addr_t ip;
    uint16_t port80;
    httpc_connection_t settings;

    IP4_ADDR(&ip31,21,236,6);
    settings.result_fn = result_fn;

    err_t err = httpc_get_file(
        &ip
        port
        "/512KB.zip"
        &settings
        rec_fn
        nullptr
        nullptr);

    if ( err != ERR_OK ) 
    {
        LogLineToUart("httpc_get_file failed (%d)"err);
    }

for now, I only log when I am receiving data:

err_t rec_fn(void *argstruct altcp_pcb *connstruct pbuf *perr_t err)
{
    LogLineToUart("sinking: %d\n"p->len);
}

void result_fn(void *arghttpc_result_t httpc_resultu32_t rx_content_lenu32_t srv_reserr_t err)
{
    LogLineToUart("download finished\n");
}

Both functions are called. So data is being received. 

The questions arise when I look at wireshark. 

1278 302.433400 192.168.1.113 185.99.25.76 TCP 58 80 → 62510 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460
1282 302.871908 185.99.25.76 192.168.1.113 HTTP 205 GET /ftp_100kb_5.log HTTP/1.1 
1283 302.872167 192.168.1.113 185.99.25.76 TCP 54 80 → 62510 [ACK] Seq=1 Ack=152 Win=29736 Len=0
1284 302.873849 192.168.1.113 185.99.25.76 TCP 526 80 → 62510 [ACK] Seq=1 Ack=152 Win=29736 Len=472 [TCP segment of a reassembled PDU]
1285 302.874100 192.168.1.113 185.99.25.76 TCP 526 80 → 62510 [ACK] Seq=473 Ack=152 Win=29736 Len=472 [TCP segment of a reassembled PDU]
1286 302.874253 192.168.1.113 185.99.25.76 TCP 526 80 → 62510 [ACK] Seq=945 Ack=152 Win=29736 Len=472 [TCP segment of a reassembled PDU]
1287 302.874394 192.168.1.113 185.99.25.76 TCP 526 80 → 62510 [ACK] Seq=1417 Ack=152 Win=29736 Len=472 [TCP segment of a reassembled PDU]
1288 302.874535 192.168.1.113 185.99.25.76 TCP 526 80 → 62510 [ACK] Seq=1889 Ack=152 Win=29736 Len=472 [TCP segment of a reassembled PDU]
1289 302.874688 192.168.1.113 185.99.25.76 TCP 526 80 → 62510 [PSH, ACK] Seq=2361 Ack=152 Win=29736 Len=472 [TCP segment of a reassembled PDU]
1290 302.874835 192.168.1.113 185.99.25.76 TCP 526 80 → 62510 [ACK] Seq=2833 Ack=152 Win=29736 Len=472 [TCP segment of a reassembled PDU]
1291 302.874975 192.168.1.113 185.99.25.76 TCP 526 80 → 62510 [ACK] Seq=3305 Ack=152 Win=29736 Len=472 [TCP segment of a reassembled PDU]
1292 302.875118 192.168.1.113 185.99.25.76 TCP 526 80 → 62510 [ACK] Seq=3777 Ack=152 Win=29736 Len=472 [TCP segment of a reassembled PDU]
1293 302.875261 192.168.1.113 185.99.25.76 TCP 526 [TCP Window Full] 80 → 62510 [ACK] Seq=4249 Ack=152 Win=29736 Len=472 [TCP segment of a reassembled PDU]
1295 303.465207 185.99.25.76 192.168.1.113 TCP 60 62510 → 80 [ACK] Seq=152 Ack=473 Win=4248 Len=0
1304 304.795607 192.168.1.113 185.99.25.76 TCP 526 [TCP Retransmission] 80 → 62510 [ACK] Seq=473 Ack=152 Win=29736 Len=472
1305 305.354254 185.99.25.76 192.168.1.113 TCP 60 [TCP Dup ACK 1295#1] 62510 → 80 [ACK] Seq=152 Ack=473 Win=4248 Len=0
1311 307.485583 192.168.1.113 185.99.25.76 TCP 526 [TCP Retransmission] 80 → 62510 [ACK] Seq=473 Ack=152 Win=29736 Len=472
1312 307.903785 185.99.25.76 192.168.1.113 TCP 60 [TCP Dup ACK 1295#2] 62510 → 80 [ACK] Seq=152 Ack=473 Win=4248 Len=0
1324 312.855604 192.168.1.113 185.99.25.76 TCP 526 [TCP Retransmission] 80 → 62510 [ACK] Seq=473 Ack=152 Win=29736 Len=472
1326 313.354377 185.99.25.76 192.168.1.113 TCP 60 [TCP Dup ACK 1295#3] 62510 → 80 [ACK] Seq=152 Ack=473 Win=4248 Len=0
1334 317.187014 185.99.25.76 192.168.1.113 TCP 60 62510 → 80 [RST, ACK] Seq=152 Ack=473 Win=28690 Len=0

The data sent by the server doesn't add up to 512KB by a long shot. And my STM doesn't seem to send acks fast enough, resulting in window full + retransmits.

I also enabled lwip debugging, just to be 100% no packets are being dropped. They are not.

I have been playing with lwipopt settings (window size, MSS, #memp,. etc) but nothing results in a better performance.

Does somebody recognize these problems from when you just started working with lwip? I am running out of ideas on how to approach this. 

For what it's worth, my lwftp implementation works for the downloading part. The traffic in wireshark is good. Data is being sent, and lwip is responding with acks. Occasionally a retransmit but that's it. I also did a quick test with mqtt. Same story, works flawless, sending publishes every half second and capable of logging published messages from broker on the same rate.

I suspect everything points to wrong lwipopts configuration. I'll post my config below, if there are any alarm bells ringing I would be very grateful if somebody would point them out to me.

lwipopts.h

/*
 * Copyright (c) 2001-2003 Swedish Institute of Computer Science.
 * All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without modification,
 * are permitted provided that the following conditions are met:
 *
 * 1. Redistributions of source code must retain the above copyright notice,
 *    this list of conditions and the following disclaimer.
 * 2. Redistributions in binary form must reproduce the above copyright notice,
 *    this list of conditions and the following disclaimer in the documentation
 *    and/or other materials provided with the distribution.
 * 3. The name of the author may not be used to endorse or promote products
 *    derived from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED
 * WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
 * MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
 * SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT
 * OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
 * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
 * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
 * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY
 * OF SUCH DAMAGE.
 *
 * This file is part of the lwIP TCP/IP stack.
 *
 * Author: Adam Dunkels <adam@sics.se>
 *
 */
#ifndef LWIP_LWIPOPTS_H
#define LWIP_LWIPOPTS_H

#ifdef LWIP_OPTTEST_FILE
#else /* LWIP_OPTTEST_FILE */

#define LWIP_IPV4                  1
#define LWIP_IPV6                  0

#define NO_SYS                     0
#define LWIP_SOCKET                (NO_SYS==0)
#define LWIP_NETCONN               (NO_SYS==0)
#define LWIP_NETIF_API             (NO_SYS==0)

#define LWIP_IGMP                  0
#define LWIP_ICMP                  0

#ifdef LWIP_UDP
#define LWIP_SNMP                  LWIP_UDP
#define MIB2_STATS                 LWIP_SNMP

#define LWIP_DNS                   LWIP_UDP
#define LWIP_MDNS_RESPONDER        LWIP_UDP

#define LWIP_NUM_NETIF_CLIENT_DATA (LWIP_MDNS_RESPONDER)
#endif


#define LWIP_HAVE_LOOPIF           0
#define LWIP_NETIF_LOOPBACK        0
#define LWIP_LOOPBACK_MAX_PBUFS    0

#define TCP_LISTEN_BACKLOG         1

#define TCPIP_MBOX_SIZE            10
#define TCPIP_THREAD_STACKSIZE     4000
#define DEFAULT_THREAD_STACKSIZE   3000
#define DEFAULT_TCP_RECVMBOX_SIZE  10

#define LWIP_COMPAT_SOCKETS        1
#define LWIP_SO_RCVTIMEO           1
#define LWIP_SO_RCVBUF             1

#define LWIP_FREERTOS_CHECK_CORE_LOCKING  1
#define LWIP_TCPIP_CORE_LOCKING           1

#define LWIP_NETIF_LINK_CALLBACK        1
#define LWIP_NETIF_STATUS_CALLBACK      1
#define LWIP_NETIF_EXT_STATUS_CALLBACK  1

#define TCPIP_THREAD_PRIO               2

// Uncomment next line if more lwip logging is needed
#define LWIP_DEBUG 

#ifdef LWIP_DEBUG

#define LWIP_DBG_MIN_LEVEL         0
#define PPP_DEBUG                  LWIP_DBG_ON
#define MEM_DEBUG                  LWIP_DBG_OFF
#define MEMP_DEBUG                 LWIP_DBG_OFF
#define PBUF_DEBUG                 LWIP_DBG_OFF
#define API_LIB_DEBUG              LWIP_DBG_OFF
#define API_MSG_DEBUG              LWIP_DBG_OFF
#define TCPIP_DEBUG                LWIP_DBG_OFF
#define NETIF_DEBUG                LWIP_DBG_OFF
#define SOCKETS_DEBUG              LWIP_DBG_OFF
#define DNS_DEBUG                  LWIP_DBG_OFF
#define AUTOIP_DEBUG               LWIP_DBG_OFF
#define DHCP_DEBUG                 LWIP_DBG_OFF
#define IP_DEBUG                   LWIP_DBG_OFF
#define IP_REASS_DEBUG             LWIP_DBG_OFF
#define ICMP_DEBUG                 LWIP_DBG_OFF
#define IGMP_DEBUG                 LWIP_DBG_OFF
#define UDP_DEBUG                  LWIP_DBG_OFF
#define TCP_DEBUG                  LWIP_DBG_OFF
#define TCP_INPUT_DEBUG            LWIP_DBG_OFF
#define TCP_OUTPUT_DEBUG           LWIP_DBG_OFF
#define TCP_RTO_DEBUG              LWIP_DBG_OFF
#define TCP_CWND_DEBUG             LWIP_DBG_OFF
#define TCP_WND_DEBUG              LWIP_DBG_OFF
#define TCP_FR_DEBUG               LWIP_DBG_OFF
#define TCP_QLEN_DEBUG             LWIP_DBG_OFF
#define TCP_RST_DEBUG              LWIP_DBG_OFF
#endif

#define LWIP_DBG_TYPES_ON         (LWIP_DBG_ON|LWIP_DBG_TRACE|LWIP_DBG_STATE|LWIP_DBG_FRESH|LWIP_DBG_HALT)


/* ---------- Memory options ---------- */
/* MEM_ALIGNMENT: should be set to the alignment of the CPU for which
   lwIP is compiled. 4 byte alignment -> define MEM_ALIGNMENT to 4, 2
   byte alignment -> define MEM_ALIGNMENT to 2. */
/* MSVC port: intel processors don't need 4-byte alignment,
   but are faster that way! */
#define MEM_ALIGNMENT           4U

#define MEMP_SANITY_CHECK       1

/**
 * MEMP_OVERFLOW_CHECK: memp overflow protection reserves a configurable 
 * amount of bytes before and after each memp element in every pool and fills 
 * it with a prominent default value. MEMP_OVERFLOW_CHECK == 0 no 
 * checking MEMP_OVERFLOW_CHECK == 1 checks each element when it is 
 * freed MEMP_OVERFLOW_CHECK >= 2 checks each element in every pool 
 * every time memp_malloc() or memp_free() is called (useful but slow!)
 */
#define MEMP_OVERFLOW_CHECK     1

/* MEM_SIZE: the size of the heap memory. If the application will send
a lot of data that needs to be copied, this should be set high. */
#define MEM_SIZE               (30*1024)

/* MEMP_NUM_PBUF: the number of memp struct pbufs. If the application
   sends a lot of data out of ROM (or other static memory), this
   should be set high. */
#define MEMP_NUM_PBUF           100
/* MEMP_NUM_RAW_PCB: the number of UDP protocol control blocks. One
   per active RAW "connection". */
#define MEMP_NUM_RAW_PCB        0
/* MEMP_NUM_UDP_PCB: the number of UDP protocol control blocks. One
   per active UDP "connection". */
#define MEMP_NUM_UDP_PCB        0
/* MEMP_NUM_TCP_PCB: the number of simulatenously active TCP
   connections. */
#define MEMP_NUM_TCP_PCB        6
/* MEMP_NUM_TCP_PCB_LISTEN: the number of listening TCP
   connections. */
#define MEMP_NUM_TCP_PCB_LISTEN 0
/* MEMP_NUM_TCP_SEG: the number of simultaneously queued TCP
   segments. */
#define MEMP_NUM_TCP_SEG        16
/* MEMP_NUM_SYS_TIMEOUT: the number of simulateously active
   timeouts. */
#define MEMP_NUM_SYS_TIMEOUT    17

/* The following four are used only with the sequential API and can be
   set to 0 if the application only will use the raw API. */
/* MEMP_NUM_NETBUF: the number of struct netbufs. */
#define MEMP_NUM_NETBUF         8
/* MEMP_NUM_NETCONN: the number of struct netconns. */
#define MEMP_NUM_NETCONN        12
/* MEMP_NUM_TCPIP_MSG_*: the number of struct tcpip_msg, which is used
   for sequential API communication and incoming packets. Used in
   src/api/tcpip.c. */
#define MEMP_NUM_TCPIP_MSG_API   16
#define MEMP_NUM_TCPIP_MSG_INPKT 16


/** SYS_LIGHTWEIGHT_PROT
 * define SYS_LIGHTWEIGHT_PROT in lwipopts.h if you want inter-task protection
 * for certain critical regions during buffer allocation, deallocation and memory
 * allocation and deallocation.
 */
#define SYS_LIGHTWEIGHT_PROT    (NO_SYS==0)

/**
 * TODO BP
 * Check opt.h for reference
 * This might be a bad idea
 */
#define ETH_PAD_SIZE             2


/* ---------- TCP options ---------- */
#define LWIP_TCP                1
#define TCP_TTL                 255

#define LWIP_ALTCP              (LWIP_TCP)
#ifdef LWIP_HAVE_MBEDTLS
#define LWIP_ALTCP_TLS          (LWIP_TCP)
#define LWIP_ALTCP_TLS_MBEDTLS  (LWIP_TCP)
#endif


/* Controls if TCP should queue segments that arrive out of
   order. Define to 0 if your device is low on memory. */
#define TCP_QUEUE_OOSEQ         1

/* TCP Maximum segment size. */
#define TCP_MSS                 (512 - 40)     /* TCP_MSS = (Ethernet MTU - IP header size - TCP header size) */

/* TCP receive window. */
#define TCP_WND                 (10*TCP_MSS)

/* TCP sender buffer space (bytes). */
#define TCP_SND_BUF             (2*TCP_MSS)

/* PBUF_POOL_SIZE: the number of buffers in the pbuf pool. */
#define PBUF_POOL_SIZE          20

/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool. */
#define PBUF_POOL_BUFSIZE       (TCP_MSS + 40 + 40)

/* TCP sender buffer space (pbufs). This must be at least = 2 *
   TCP_SND_BUF/TCP_MSS for things to work. */
#define TCP_SND_QUEUELEN        (4* TCP_SND_BUF/TCP_MSS)

/* TCP writable space (bytes). This must be less than or equal
   to TCP_SND_BUF. It is the amount of space which must be
   available in the tcp snd_buf for select to return writable */
#define TCP_SNDLOWAT            (TCP_SND_BUF/2)

/* Maximum number of retransmissions of data segments. */
#define TCP_MAXRTX              12

/* Maximum number of retransmissions of SYN segments. */
#define TCP_SYNMAXRTX           4

/* ---------- ARP options ---------- */
#define LWIP_ARP                1
#define ARP_TABLE_SIZE          10
#define ARP_QUEUEING            1


/* ---------- IP options ---------- */
/* Define IP_FORWARD to 1 if you wish to have the ability to forward
   IP packets across network interfaces. If you are going to run lwIP
   on a device with only one network interface, define this to 0. */
#define IP_FORWARD              0

/* IP reassembly and segmentation.These are orthogonal even
 * if they both deal with IP fragments */
#define IP_REASSEMBLY           1
#define IP_REASS_MAX_PBUFS      (10 * ((1500 + PBUF_POOL_BUFSIZE - 1/ PBUF_POOL_BUFSIZE))
#define MEMP_NUM_REASSDATA      IP_REASS_MAX_PBUFS
#define IP_FRAG                 1
#define IPV6_FRAG_COPYHEADER    1

/* ---------- ICMP options ---------- */
#define ICMP_TTL                255


/* ---------- DHCP options ---------- */
/* Define LWIP_DHCP to 1 if you want DHCP configuration of
   interfaces. */
#define LWIP_DHCP               LWIP_UDP

/* 1 if you want to do an ARP check on the offered address
   (recommended). */
#define DHCP_DOES_ARP_CHECK    (LWIP_DHCP)


/* ---------- AUTOIP options ------- */
#define LWIP_AUTOIP            (LWIP_DHCP)
#define LWIP_DHCP_AUTOIP_COOP  (LWIP_DHCP && LWIP_AUTOIP)


/* ---------- UDP options ---------- */
#define LWIP_UDP                0
#define LWIP_UDPLITE            LWIP_UDP
#define UDP_TTL                 255


/* ---------- RAW options ---------- */
#define LWIP_RAW                0

/* ---------- Statistics options ---------- */

#define LWIP_STATS              1
#define LWIP_STATS_DISPLAY      1

#if LWIP_STATS
#define LINK_STATS              0
#define IP_STATS                0
#define ICMP_STATS              0
#define IGMP_STATS              0
#define IPFRAG_STATS            0
#define UDP_STATS               0
#define TCP_STATS               0
#define MEM_STATS               1
#define MEMP_STATS              1
#define PBUF_STATS              0
#define SYS_STATS               1
#endif /* LWIP_STATS */

/* ---------- NETBIOS options ---------- */
#define LWIP_NETBIOS_RESPOND_NAME_QUERY 0

/* ---------- PPP options ---------- */

#define PPP_SUPPORT             1      /* Set > 0 for PPP */

#if PPP_SUPPORT

#define PRINTPKT_SUPPORT        0      // TODO BP, added for debugging
#define NUM_PPP                 1      /* Max PPP sessions. */
#define PPP_NOTIFY_PHASE        1      // TODO BP, added for debugging

/* Select modules to enable.  Ideally these would be set in the makefile but
 * we're limited by the command line length so you need to modify the settings
 * in this file.
 */
#define PPPOE_SUPPORT           0
#define PPPOS_SUPPORT           1

#define PAP_SUPPORT             0      /* Set > 0 for PAP. */
#define CHAP_SUPPORT            0      /* Set > 0 for CHAP. */
#define MSCHAP_SUPPORT          0      /* Set > 0 for MSCHAP */
#define CBCP_SUPPORT            0      /* Set > 0 for CBCP (NOT FUNCTIONAL!) */
#define CCP_SUPPORT             0      /* Set > 0 for CCP */
#define VJ_SUPPORT              0      /* Set > 0 for VJ header compression. */
#define MD5_SUPPORT             0      /* Set > 0 for MD5 (see also CHAP) */

#endif /* PPP_SUPPORT */

#endif /* LWIP_OPTTEST_FILE */

/* The following defines must be done even in OPTTEST mode: */

#if !defined(NO_SYS|| !NO_SYS /* default is 0 */
void sys_check_core_locking(void);
#define LWIP_ASSERT_CORE_LOCKED()  sys_check_core_locking()
#endif

#endif /* LWIP_LWIPOPTS_H */







































reply via email to

[Prev in Thread] Current Thread [Next in Thread]