Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - override mapred-site.xml from command line


Copy link to this message
-
Re: override mapred-site.xml from command line
Marcos Ortiz 2012-06-07, 13:16


On 06/06/2012 07:44 PM, Sid Kumar wrote:
> I am able to set it via the API.
> Configuration.setBoolean(mapred.output.compress,true). This works!
>
> But the -D from the command line still doesn't work. Any idea what I
> may be missing here?
>
> Some additional info - Also when I try running the -D on command line
> on a local cluster (pseudo distributed mode) it works, but when I try
> it on a fully distributed cluster running jobs from a client machine
> it doesn't work. Is there a different way for setting it in this case
> - in hadoop-env perhaps?
>
> Thanks
> Sid
>
> On Wed, Jun 6, 2012 at 4:06 PM, Sid Kumar <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     Mayank,
>     I dont have a final tag for that property set. I looked at the
>     mapred-default.xml in the src/mapred folder and that doesn't have
>     a final tag too. Should I set it explicitly to false?
>
You should do it explicitly.
You should read the excellent blog post from Lars Francke where he did a
great job explaining parameter by parameter and why is recommendable to
set them to final.
http://gbif.blogspot.com/2011/01/setting-up-hadoop-cluster-part-1-manual.html

Regards
>
>
>     Sid
>
>
>     On Wed, Jun 6, 2012 at 3:50 PM, Mayank Bansal <[EMAIL PROTECTED]
>     <mailto:[EMAIL PROTECTED]>> wrote:
>
>         Check your mapred site xml if these parameters have
>         <final>true</final>
>
>         making final to false should solve your problem.
>
>
>         On Wed, Jun 6, 2012 at 3:41 PM, Sid Kumar <[EMAIL PROTECTED]
>         <mailto:[EMAIL PROTECTED]>> wrote:
>
>             Hi,
>             I am trying to override mapred-site.xml (more specifically
>             mapred.compress.map.output
>             and mapred.output.compression.
>             codec) from the command line when I
>             execute the jar.
>             I have been using hadoop jar <jarname> <class> -
>             Dmapred.compress.map.output=true and
>             -Dmapred.output.compression.codec=org.apache.hadoop.io.SnappyCodec
>
>
>             The above doesnt work as the job.xml for the jar still
>             uses the default properties and not the one i specify
>             here. Is there a different approach to override these
>             properties. I am submitting jobs from a client machine
>             that has the same version of configuration files as my
>             cluster.
>
>             Thanks
>
>             Sid
>
>
>
>

--
Marcos Luis Ort�z Valmaseda
  Data Engineer&&  Sr. System Administrator at UCI
  http://marcosluis2186.posterous.com
  http://www.linkedin.com/in/marcosluis2186
  Twitter: @marcosluis2186

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci