|
From: | Markus Schiltknecht |
Subject: | Re: [Monotone-devel] Re: zlib vs gzip |
Date: | Tue, 30 Oct 2007 13:48:45 +0100 |
User-agent: | Mozilla-Thunderbird 2.0.0.6 (X11/20071008) |
Hi, Jack Lloyd wrote:
Yes with that assumption I'd have to agree, but on anything else (eg, 4-way Opterons, old Unix machines with poor CPU and relatively powerful I/O, etc) it probably hurts quite a bit. Even on my Core2 (with a RAID-1 setup, so each write has to hit all three disks), it is faster (wall clock) to write 200 Mb to disk than it is to first compress it with gzip and write the 30 Mb compressed result to disk.
Hm.. wow, yeah, on a very quick test, I got ~ 170 MB/s plain throughput, but only 13.5 MB/s when piping through gzip. Is compression really that slow even on modern hardware? I thought that changed long ago, well...
Thanks for pointing that out.I remember the LZMA benchmarks [1], those seem to be more representative than my quick test. They list the following throughputs for an AMD mobile Athlon XP2400+ (in MB/sec):
compression decompression compressed size (%) gzip -1 18 61 40,6 gzip -6 8.3 63 37.2 gzip -9 3.0 65 37.0 bzip2 -9 1.3 5.3 34.0 lzmash -9 < 0.27 20 25.4Thus, not even gzip -1 compression is close to a harddrive's throughput. Only gzip decompression comes somewhat close, probably making it quite a good fit.
I'm reminded of TOAST in PostgreSQL. They use a fast variant of an LZ compression algorithm. I'm wondering what throughput that achieves.
Regards Markus [1]: some LZMA benchmarks: http://tukaani.org/lzma/benchmarks
[Prev in Thread] | Current Thread | [Next in Thread] |