Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Yarn job runs in Local Mode even though the cluster is running in Distributed Mode


Copy link to this message
-
Yarn job runs in Local Mode even though the cluster is running in Distributed Mode
Hi All

I am using cdh4 for running a HBase cluster on CentOs6.0. I have 5
nodes in my cluster(2 Admin Node and 3 DN).
My resourcemanager is up and running and showing that all three DN are
running the nodemanager. HDFS is also working fine and showing 3 DN's.

But when i fire the pi example job. It starts to run in Local mode.
Here is the console output:
sudo -u hdfs yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-
examples.jar pi 10 1000000000
Number of Maps  = 10
Samples per Map = 1000000000
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Wrote input for Map #5
Wrote input for Map #6
Wrote input for Map #7
Wrote input for Map #8
Wrote input for Map #9
Starting Job
12/06/13 12:03:27 WARN conf.Configuration: session.id is deprecated.
Instead, use dfs.metrics.session-id
12/06/13 12:03:27 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId12/06/13 12:03:27 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
12/06/13 12:03:27 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
12/06/13 12:03:28 INFO mapred.FileInputFormat: Total input paths to
process : 10
12/06/13 12:03:29 INFO mapred.JobClient: Running job: job_local_0001
12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter set in
config null
12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter is
org.apache.hadoop.mapred.FileOutputCommitter
12/06/13 12:03:29 WARN mapreduce.Counters: Group
org.apache.hadoop.mapred.Task$Counter is deprecated. Use
org.apache.hadoop.mapreduce.TaskCounter instead
12/06/13 12:03:29 INFO util.ProcessTree: setsid exited with exit code
0
12/06/13 12:03:29 INFO mapred.Task:  Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3d46e381
12/06/13 12:03:29 WARN mapreduce.Counters: Counter name
MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
name and  BYTES_READ as counter name instead
12/06/13 12:03:29 INFO mapred.MapTask: numReduceTasks: 1
12/06/13 12:03:29 INFO mapred.MapTask: io.sort.mb = 100
12/06/13 12:03:30 INFO mapred.MapTask: data buffer = 79691776/99614720
12/06/13 12:03:30 INFO mapred.MapTask: record buffer = 262144/327680
12/06/13 12:03:30 INFO mapred.JobClient:  map 0% reduce 0%
12/06/13 12:03:35 INFO mapred.LocalJobRunner: Generated 95735000
samples.
12/06/13 12:03:36 INFO mapred.JobClient:  map 100% reduce 0%
12/06/13 12:03:38 INFO mapred.LocalJobRunner: Generated 151872000
samples.

Here is the content of yarn-site.xml:

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce.shuffle</value>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>

  <property>
    <description>List of directories to store localized files in.</
description>
    <name>yarn.nodemanager.local-dirs</name>
    <value>/disk/yarn/local</value>
  </property>

  <property>
    <description>Where to store container logs.</description>
    <name>yarn.nodemanager.log-dirs</name>
    <value>/disk/yarn/logs</value>
  </property>

  <property>
    <description>Where to aggregate logs to.</description>
    <name>yarn.nodemanager.remote-app-log-dir</name>
    <value>/var/log/hadoop-yarn/apps</value>
  </property>

  <property>
    <description>Classpath for typical applications.</description>
     <name>yarn.application.classpath</name>
     <value>
        $HADOOP_CONF_DIR,
        $HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
        $HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
        $HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
        $YARN_HOME/*,$YARN_HOME/lib/*
     </value>
  </property>
<property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>ihub-an-g1:8025</value>
</property>
<property>
        <name>yarn.resourcemanager.address</name>
        <value>ihub-an-g1:8040</value>
</property>
<property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>ihub-an-g1:8030</value>
</property>
<property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>ihub-an-g1:8141</value>
</property>
<property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>ihub-an-g1:8088</value>
</property>
<property>
        <name>mapreduce.jobhistory.intermediate-done-dir</name>
        <value>/disk/mapred/jobhistory/intermediate/done</value>
</property>
<property>
        <name>>mapreduce.jobhistory.done-dir</name>
        <value>/disk/mapred/jobhistory/done</value>
</property>
</configuration>

Can anyone tell me what is the problem over here? Appreciate your
help.
Thanks,
Anil Gupta
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB