[bug #60106] wget mistakenly interprets Content-Range as bytes, even with Range-Unit: items, and truncates content

[Andreas Franke]

URL:
  <https://savannah.gnu.org/bugs/?60106>

                 Summary: wget mistakenly interprets Content-Range as bytes,
even with Range-Unit: items, and truncates content
                 Project: GNU Wget
            Submitted by: afranke
            Submitted on: Tue 23 Feb 2021 07:10:59 PM UTC
                Category: Program Logic
                Severity: 3 - Normal
                Priority: 5 - Normal
                  Status: None
                 Privacy: Public
             Assigned to: None
         Originator Name: 
        Originator Email: 
             Open/Closed: Open
                 Release: 1.19.4
         Discussion Lock: Any
        Operating System: GNU/Linux
         Reproducibility: Every Time
           Fixed Release: None
         Planned Release: None
              Regression: No
           Work Required: None
          Patch Included: None

    _______________________________________________________

Details:

When a server sends the following response headers:

  Accept-Ranges: items
  Range-Unit: items
  Content-Range: 0-110/111

but *no* 'Content-Length' response header, then wget says

Length: 111 [...]

and truncates the response content after 111 bytes (or, when using https,
after a few KB), even though the full response content is much larger (e.g.
720 KB).

What's worse, --ignore-length doesn't help with this at all.

While a range unit of "bytes" is the most common case, IMO wget should not
assume a range unit of "bytes" when the 'Range-Unit' header says something
else.

To aggravate the problem even further, wget then retries with a range request
for the remaining content using a bytes range, even though the server just
stated it does NOT support bytes ranges, but only items ranges.  (This latest
problem can be avoided by disallowing retries with the -t1 option.)

This issue can occur e.g. with a REST api providing lists of items as
application/json content.




    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?60106>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/