-Re: Can spill to disk be in compressed format to reduce I/O?
bejoy.hadoop@... 2012-01-12, 15:49
Is map output compression enabled?
The config param would be like
(It is from my memory, Please cross check)
From: Frank Grimes
To: [EMAIL PROTECTED]
ReplyTo: [EMAIL PROTECTED]
Subject: Can spill to disk be in compressed format to reduce I/O?
Sent: Jan 12, 2012 21:10
We're trying to speed up an M/R job which combines multiple .avro files.
We've noticed that when it spills to disk, it's in uncompressed format.
Is there a way to make it spill temporary segments as .avro with Deflate compression?
Bejoy K S