Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> RE: Map output compression in Hadoop 1.0.3

Copy link to this message
RE: Map output compression in Hadoop 1.0.3

Hi Andy and list,

Apologies - I've not been looking at my list inbox for a while, so missed this request. I'm running some tests as I type, and will report back when they're done. I'm running the same job for bzip2, gzip and snappy codecs versus no map out compression. I guess I should include LZO in the comparison too, but the codec wasn't obvious in the o.a.h.io.compress.* areas of Hadoop. If someone could point out where to find this codec, that'd be really handy. If not I could always google it :)


From: Kartashov, Andy [[EMAIL PROTECTED]]
Sent: 29 November 2012 16:09
Subject: RE: Map output compression in Hadoop 1.0.3


Can you please share with us on the permorfmance improvement (if any) after using compression in map.output? I was abpout to start looking into it myself.   What compression codec did you use?


-----Original Message-----
From: Tony Burton [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, November 28, 2012 6:38 AM
Subject: RE: Map output compression in Hadoop 1.0.3

Also, another point that prompted my initial question: I'd come across "mapred.compress.map.output" in the documentation, but I wasn't 100% sure if there has been or will be any equivalence or correspondence between config setting like this one and the naming of the stable and new API.

For example, we've got o.a.h.mapreduce.Job rather than o.a.h.mapred.JobConf as previously mentioned, from the "mapred" and "mapreduce" parts of the API.

Are config settings that begin with mapred.* related to the stable API with the implication that there's an mapreduce.* equivalent (eg mapred.compress.map.output vs mapreduce.compress.map.output), or am I seeing a connection that doesn't exist?

(Hope that makes sense!)
-----Original Message-----
From: Harsh J [mailto:[EMAIL PROTECTED]]
Sent: 28 November 2012 11:25
Subject: Re: Map output compression in Hadoop 1.0.3


The property mapred.output.compress, as its name reads, controls job-output compression, not intermediate/transient data compression, which is what you mean by "Map output compression".

Also note that this property is a per job one and can be toggled, if a user wanted, on/off for each job specifically.

These should be the many ways, exhaustively, for MR1, to turn on "Map output compression":

1. Set "mapred.compress.map.output" to true in your client's mapred-site.xml to turn it on for all jobs run from such a client machine.
2. Set the above in cluster, with <final>true</final> at every node (JT plus TTs) and restart them, to turn it on for all job, regardless of what the job itself specifies.
3. Turn it on per-job basis:
3.1. Stable API: JobConf.setCompressMapOutput(true);
3.2. New API: Job.getConfiguration().set("mapred.compress.map.output", true);

On Wed, Nov 28, 2012 at 4:42 PM, Tony Burton <[EMAIL PROTECTED]> wrote:
> Hi,
> Quick question: What's the best way to turn on Map Output Compression
> in Hadoop 1.0.3? The tutorial at
> http://hadoop.apache.org/docs/r1.0.3/mapred_tutorial.html says to use
> JobConf.setCompressMapOutput(boolean), but I'm using
> o.a.h.mapreduce.Job rather than o.a.h.mapred.JobConf.
> Is it simply a case of using getConf.set("mapred.output.compress",
> true) then constructing my Job from the Configuration object, or is
> there more direct way that I've missed?
> Thanks,
> Tony
> **********************************************************************
> ******* P Please consider the environment before printing this email
> or attachments
> This email and any attachments are confidential, protected by
> copyright and may be legally privileged. If you are not the intended
> recipient, then the dissemination or copying of this email is
> prohibited. If you have received this in error, please notify the
> sender by replying by email and then delete the email completely from

Harsh J
Please consider the environment before printing this email

Inbound Email has been scanned for viruses and SPAM
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel