Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Can spill to disk be in compressed format to reduce I/O?


Copy link to this message
-
Re: Can spill to disk be in compressed format to reduce I/O?
Hi Frank
       Is map output compression enabled?

The config param would be like
mapred.map.output.compress=true
(It is from my memory, Please cross check)

------Original Message------
From: Frank Grimes
To: [EMAIL PROTECTED]
ReplyTo: [EMAIL PROTECTED]
Subject: Can spill to disk be in compressed format to reduce I/O?
Sent: Jan 12, 2012 21:10

Hi All,

We're trying to speed up an M/R job which combines multiple .avro files.
We've noticed that when it spills to disk, it's in uncompressed format.
Is there a way to make it spill temporary segments as .avro with Deflate compression?

Thanks,

Frank Grimes

Regards
Bejoy K S
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB