-Re: Can spill to disk be in compressed format to reduce I/O?
Arun C Murthy 2012-01-12, 23:15
Temporary map-ouput files don't use Avro format. There is a custom format which should be compressed if you set mapred.compress.map.output.
On Jan 12, 2012, at 8:08 AM, Frank Grimes wrote:
> I tried conf.setBoolean("mapred.compress.map.output", true); but it didn't seem to work.
> Also, since I'm using the Avro mapred APIs, maybe there's something Avro specific to get it enabled?
> Should I ask on the Avro mailing lists?
> Frank Grimes
> On 2012-01-12, at 10:49 AM, [EMAIL PROTECTED] wrote:
>> Hi Frank
>> Is map output compression enabled?
>> The config param would be like
>> (It is from my memory, Please cross check)
>> ------Original Message------
>> From: Frank Grimes
>> To: [EMAIL PROTECTED]
>> ReplyTo: [EMAIL PROTECTED]
>> Subject: Can spill to disk be in compressed format to reduce I/O?
>> Sent: Jan 12, 2012 21:10
>> Hi All,
>> We're trying to speed up an M/R job which combines multiple .avro files.
>> We've noticed that when it spills to disk, it's in uncompressed format.
>> Is there a way to make it spill temporary segments as .avro with Deflate compression?
>> Frank Grimes
>> Bejoy K S