Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> override mapred-site.xml from command line


Copy link to this message
-
Re: override mapred-site.xml from command line


On 06/06/2012 07:44 PM, Sid Kumar wrote:
> I am able to set it via the API.
> Configuration.setBoolean(mapred.output.compress,true). This works!
>
> But the -D from the command line still doesn't work. Any idea what I
> may be missing here?
>
> Some additional info - Also when I try running the -D on command line
> on a local cluster (pseudo distributed mode) it works, but when I try
> it on a fully distributed cluster running jobs from a client machine
> it doesn't work. Is there a different way for setting it in this case
> - in hadoop-env perhaps?
>
> Thanks
> Sid
>
> On Wed, Jun 6, 2012 at 4:06 PM, Sid Kumar <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     Mayank,
>     I dont have a final tag for that property set. I looked at the
>     mapred-default.xml in the src/mapred folder and that doesn't have
>     a final tag too. Should I set it explicitly to false?
>
You should do it explicitly.
You should read the excellent blog post from Lars Francke where he did a
great job explaining parameter by parameter and why is recommendable to
set them to final.
http://gbif.blogspot.com/2011/01/setting-up-hadoop-cluster-part-1-manual.html

Regards
>
>
>     Sid
>
>
>     On Wed, Jun 6, 2012 at 3:50 PM, Mayank Bansal <[EMAIL PROTECTED]
>     <mailto:[EMAIL PROTECTED]>> wrote:
>
>         Check your mapred site xml if these parameters have
>         <final>true</final>
>
>         making final to false should solve your problem.
>
>
>         On Wed, Jun 6, 2012 at 3:41 PM, Sid Kumar <[EMAIL PROTECTED]
>         <mailto:[EMAIL PROTECTED]>> wrote:
>
>             Hi,
>             I am trying to override mapred-site.xml (more specifically
>             mapred.compress.map.output
>             and mapred.output.compression.
>             codec) from the command line when I
>             execute the jar.
>             I have been using hadoop jar <jarname> <class> -
>             Dmapred.compress.map.output=true and
>             -Dmapred.output.compression.codec=org.apache.hadoop.io.SnappyCodec
>
>
>             The above doesnt work as the job.xml for the jar still
>             uses the default properties and not the one i specify
>             here. Is there a different approach to override these
>             properties. I am submitting jobs from a client machine
>             that has the same version of configuration files as my
>             cluster.
>
>             Thanks
>
>             Sid
>
>
>
>

--
Marcos Luis Ort�z Valmaseda
  Data Engineer&&  Sr. System Administrator at UCI
  http://marcosluis2186.posterous.com
  http://www.linkedin.com/in/marcosluis2186
  Twitter: @marcosluis2186

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB