Re: [maintenance] Compressed JSON files and served file extension?

[Attila Lendvai]

TL;DR the filename shouldn't contain the .gz extension, and the HTTP standard 
is crap ("If no Accept-Encoding field is present in a request, the server MAY 
assume that the client will accept any content coding.").

use curl --compressed

the details:

the Content-Encoding response header instructs the client on how to decode the 
__transfer payload__ that the server is serving.

i.e. proper HTTP clients should automatically decode the content as instructed 
by the Content-Encoding response header, or at the very least warn that they do 
not understand the response encoding...

but that should not happen, because the HTTP request can contain an 
Accept-Encoding header that tells the server what the client understands, and 
it defaults to unprocessed raw data ('identity')...

except that the standard allows the server to ignore the Accept-Encoding 
request header.

well, this is the theory, but both wget and curl don't automatically decode the 
content. curl at least can be instructed to do so, which arguably should be its 
default:

curl --compressed https://guix.gnu.org/sources.json | less

--verbose can be used to inspect the reques/response headers (printed to 
stderr):

curl --verbose https://guix.gnu.org/sources.json >/dev/null
curl --verbose --compressed https://guix.gnu.org/sources.json >/dev/null

here's a detailed discussion of this very question:

https://stackoverflow.com/questions/8364640/how-to-properly-handle-a-gzipped-page-when-using-curl

so, in an ideal world wget and curl should transparently decode the content 
according to the Content-Encoding response header, and nginx should not respond 
with a compressed content when the client is not sending an Accept-Encoding 
request header.

the pragmatic solution is to use curl --compressed in scripts, and/or add it to 
your ~/.curlrc:

# to automatically decode responses with some of
# the supported Content-Encoding
compressed

HTH,

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“We need people in our lives with whom we can be as open as possible. To have 
real conversations with people may seem like such a simple, obvious suggestion, 
but it involves courage and risk.”
        — Thomas Moore (1940–)