Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> AvroStorage compression ratio


Copy link to this message
-
AvroStorage compression ratio
Based on AvroStorage code and documentation, it looks like compression
is enabled by default, codec set to "deflate". But the file size is
almost same as that of uncompressed tab separated text data.

This is probably a bug in AvroStorage, but I wanted to check if this is
somehow expected, before I open a jira to track it.

Uncompressed txt     2.12 GB
avro (default compression)    2.09 GB
avro + snappy compression     2.09 GB
lzo compressed txt      0.69 GB
Thanks,
Thejas
+
Ruslan Al-Fakikh 2012-10-22, 13:02
+
Thejas Nair 2012-10-22, 22:51
+
Ruslan Al-Fakikh 2012-10-23, 13:31
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB