[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: wget2 | Multithreaded download: Change server probing behavior to be
|
From: |
@rockdaboot |
|
Subject: |
Re: wget2 | Multithreaded download: Change server probing behavior to better match browsers (#626) |
|
Date: |
Sun, 19 Mar 2023 17:56:58 +0000 |
Tim Rühsen commented:
> I've seen servers that take advantage of it as a form of anti-scraping.
Can you provide an example ?
It sounds like a cat and mouse game. There are many different ways of doing
anti-scraping. I assume the servers will change their method after most clients
adopted a work-around for the 404 on HEAD requests and the Range requirement. I
would also assume that some servers require a Range header to be *not* set
and/or require a HEAD request *before* the GET.
So if we work-around this server behavior, I'd suggest to enable this via a new
command line option.
--
Reply to this email directly or view it on GitLab:
https://gitlab.com/gnuwget/wget2/-/issues/626#note_1319897696
You're receiving this email because of your account on gitlab.com.