Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Can spill to disk be in compressed format to reduce I/O?


+
Frank Grimes 2012-01-12, 15:40
+
bejoy.hadoop@... 2012-01-12, 15:49
+
Frank Grimes 2012-01-12, 16:08
Copy link to this message
-
Re: Can spill to disk be in compressed format to reduce I/O?
Temporary map-ouput files don't use Avro format. There is a custom format which should be compressed if you set mapred.compress.map.output.

Arun

On Jan 12, 2012, at 8:08 AM, Frank Grimes wrote:

> I tried conf.setBoolean("mapred.compress.map.output", true); but it didn't seem to work.
>
> Also, since I'm using the Avro mapred APIs, maybe there's something Avro specific to get it enabled?
> Should I ask on the Avro mailing lists?
>
> Thanks,
>
> Frank Grimes
>
>
> On 2012-01-12, at 10:49 AM, [EMAIL PROTECTED] wrote:
>
>> Hi Frank
>>      Is map output compression enabled?
>>
>> The config param would be like
>> mapred.map.output.compress=true
>> (It is from my memory, Please cross check)
>>
>> ------Original Message------
>> From: Frank Grimes
>> To: [EMAIL PROTECTED]
>> ReplyTo: [EMAIL PROTECTED]
>> Subject: Can spill to disk be in compressed format to reduce I/O?
>> Sent: Jan 12, 2012 21:10
>>
>> Hi All,
>>
>> We're trying to speed up an M/R job which combines multiple .avro files.
>> We've noticed that when it spills to disk, it's in uncompressed format.
>> Is there a way to make it spill temporary segments as .avro with Deflate compression?
>>
>> Thanks,
>>
>> Frank Grimes
>>
>> Regards
>> Bejoy K S
>
+
Tim Broberg 2012-01-12, 23:25
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB