Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Comparison between Gzip and LZO


Copy link to this message
-
Re: Comparison between Gzip and LZO
Question: Are you 100% sure that nothing else was running on that
system during the tests?
No cron jobs, no "makewhatis" or "updatedb"?

P.S. There is a permission issue with downloading one of the files.

2011/3/2 José Vinícius Pimenta Coletto <[EMAIL PROTECTED]>:
> Hi,
>
> I'm making a comparison between the following compression methods: gzip
> and lzo provided by Hadoop and gzip from package java.util.zip.
> The test consists of compression and decompression of approximately 92,000
> files with an average size of 2kb, however the decompression time of lzo is
> twice the decompression time of gzip provided by Hadoop, it does not seem
> right.
> The results obtained in the test are:
>
>      Method         |   Bytes   |               Compression
>       |                    Decompression
>         -           |     -     | Total Time(with i/o)  Time     Speed
>        | Total Time(with i/o)  Time      Speed
> Gzip (Haddop)        | 200876304 | 121.454s              43.167s
>  4,653,424.079 B/s | 332.305s              111.806s   1,796,635.326 B/s
> Lzo                  | 200876304 | 120.564s              54.072s
>  3,714,914.621 B/s | 509.371s              184.906s   1,086,368.904 B/s
> Gzip (java.util.zip) | 200876304 | 148.014s              63.414s
>  3,167,647.371 B/s | 483.148s              4.528s    44,360,682.244 B/s
>
> You can see the code I'm using to the test here:
> http://www.linux.ime.usp.br/~jvcoletto/compression/
>
> Can anyone explain me why am I getting these results?
> Thanks.
>

--
Met vriendelijke groeten,

Niels Basjes
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB