Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Can spill to disk be in compressed format to reduce I/O?


Copy link to this message
-
Re: Can spill to disk be in compressed format to reduce I/O?
Hi Frank
       Is map output compression enabled?

The config param would be like
mapred.map.output.compress=true
(It is from my memory, Please cross check)

------Original Message------
From: Frank Grimes
To: [EMAIL PROTECTED]
ReplyTo: [EMAIL PROTECTED]
Subject: Can spill to disk be in compressed format to reduce I/O?
Sent: Jan 12, 2012 21:10

Hi All,

We're trying to speed up an M/R job which combines multiple .avro files.
We've noticed that when it spills to disk, it's in uncompressed format.
Is there a way to make it spill temporary segments as .avro with Deflate compression?

Thanks,

Frank Grimes

Regards
Bejoy K S