Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Yarn job runs in Local Mode even though the cluster is running in Distributed Mode


Copy link to this message
-
Re: Yarn job runs in Local Mode even though the cluster is running in Distributed Mode
According to the CDH 4 official documentation, you should install a
JobHistory server for your MRv2 (YARN)
cluster.
https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster

How to configure the HistoryServer
https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster#DeployingMapReducev2%28YARN%29onaCluster-Step3

On 06/13/2012 03:16 PM, anil gupta wrote:
> Hi All
>
> I am using cdh4 for running a HBase cluster on CentOs6.0. I have 5
> nodes in my cluster(2 Admin Node and 3 DN).
> My resourcemanager is up and running and showing that all three DN are
> running the nodemanager. HDFS is also working fine and showing 3 DN's.
>
> But when i fire the pi example job. It starts to run in Local mode.
> Here is the console output:
> sudo -u hdfs yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-
> examples.jar pi 10 1000000000
> Number of Maps  = 10
> Samples per Map = 1000000000
> Wrote input for Map #0
> Wrote input for Map #1
> Wrote input for Map #2
> Wrote input for Map #3
> Wrote input for Map #4
> Wrote input for Map #5
> Wrote input for Map #6
> Wrote input for Map #7
> Wrote input for Map #8
> Wrote input for Map #9
> Starting Job
> 12/06/13 12:03:27 WARN conf.Configuration: session.id is deprecated.
> Instead, use dfs.metrics.session-id
> 12/06/13 12:03:27 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId> 12/06/13 12:03:27 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 12/06/13 12:03:27 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the
> same.
> 12/06/13 12:03:28 INFO mapred.FileInputFormat: Total input paths to
> process : 10
> 12/06/13 12:03:29 INFO mapred.JobClient: Running job: job_local_0001
> 12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter set in
> config null
> 12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter is
> org.apache.hadoop.mapred.FileOutputCommitter
> 12/06/13 12:03:29 WARN mapreduce.Counters: Group
> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> org.apache.hadoop.mapreduce.TaskCounter instead
> 12/06/13 12:03:29 INFO util.ProcessTree: setsid exited with exit code
> 0
> 12/06/13 12:03:29 INFO mapred.Task:  Using ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3d46e381
> 12/06/13 12:03:29 WARN mapreduce.Counters: Counter name
> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
> name and  BYTES_READ as counter name instead
> 12/06/13 12:03:29 INFO mapred.MapTask: numReduceTasks: 1
> 12/06/13 12:03:29 INFO mapred.MapTask: io.sort.mb = 100
> 12/06/13 12:03:30 INFO mapred.MapTask: data buffer = 79691776/99614720
> 12/06/13 12:03:30 INFO mapred.MapTask: record buffer = 262144/327680
> 12/06/13 12:03:30 INFO mapred.JobClient:  map 0% reduce 0%
> 12/06/13 12:03:35 INFO mapred.LocalJobRunner: Generated 95735000
> samples.
> 12/06/13 12:03:36 INFO mapred.JobClient:  map 100% reduce 0%
> 12/06/13 12:03:38 INFO mapred.LocalJobRunner: Generated 151872000
> samples.
>
> Here is the content of yarn-site.xml:
>
> <configuration>
>    <property>
>      <name>yarn.nodemanager.aux-services</name>
>      <value>mapreduce.shuffle</value>
>    </property>
>
>    <property>
>      <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>      <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>    </property>
>
>    <property>
>      <name>yarn.log-aggregation-enable</name>
>      <value>true</value>
>    </property>
>
>    <property>
>      <description>List of directories to store localized files in.</
> description>
>      <name>yarn.nodemanager.local-dirs</name>
>      <value>/disk/yarn/local</value>
>    </property>
>
>    <property>
>      <description>Where to store container logs.</description>
>      <name>yarn.nodemanager.log-dirs</name>
>      <value>/disk/yarn/logs</value>
>    </property>
>
>    <property>
>      <description>Where to aggregate logs to.</description>

Marcos Luis Ort�z Valmaseda
  Data Engineer&&  Sr. System Administrator at UCI
  http://marcosluis2186.posterous.com
  http://www.linkedin.com/in/marcosluis2186
  Twitter: @marcosluis2186
10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci