Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Yarn job runs in Local Mode even though the cluster is running in Distributed Mode


Copy link to this message
-
Re: Yarn job runs in Local Mode even though the cluster is running in Distributed Mode
Forgot to mention:
Hadoop version: Hadoop 2.0.0-cdh4.0.0

On Wed, Jun 13, 2012 at 12:16 PM, anil gupta <[EMAIL PROTECTED]> wrote:

> Hi All
>
> I am using cdh4 for running a HBase cluster on CentOs6.0. I have 5
> nodes in my cluster(2 Admin Node and 3 DN).
> My resourcemanager is up and running and showing that all three DN are
> running the nodemanager. HDFS is also working fine and showing 3 DN's.
>
> But when i fire the pi example job. It starts to run in Local mode.
> Here is the console output:
> sudo -u hdfs yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-
> examples.jar pi 10 1000000000
> Number of Maps  = 10
> Samples per Map = 1000000000
> Wrote input for Map #0
> Wrote input for Map #1
> Wrote input for Map #2
> Wrote input for Map #3
> Wrote input for Map #4
> Wrote input for Map #5
> Wrote input for Map #6
> Wrote input for Map #7
> Wrote input for Map #8
> Wrote input for Map #9
> Starting Job
> 12/06/13 12:03:27 WARN conf.Configuration: session.id is deprecated.
> Instead, use dfs.metrics.session-id
> 12/06/13 12:03:27 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId> 12/06/13 12:03:27 INFO util.NativeCodeLoader: Loaded the native-hadoop
> library
> 12/06/13 12:03:27 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the
> same.
> 12/06/13 12:03:28 INFO mapred.FileInputFormat: Total input paths to
> process : 10
> 12/06/13 12:03:29 INFO mapred.JobClient: Running job: job_local_0001
> 12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter set in
> config null
> 12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter is
> org.apache.hadoop.mapred.FileOutputCommitter
> 12/06/13 12:03:29 WARN mapreduce.Counters: Group
> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> org.apache.hadoop.mapreduce.TaskCounter instead
> 12/06/13 12:03:29 INFO util.ProcessTree: setsid exited with exit code
> 0
> 12/06/13 12:03:29 INFO mapred.Task:  Using ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3d46e381
> 12/06/13 12:03:29 WARN mapreduce.Counters: Counter name
> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
> name and  BYTES_READ as counter name instead
> 12/06/13 12:03:29 INFO mapred.MapTask: numReduceTasks: 1
> 12/06/13 12:03:29 INFO mapred.MapTask: io.sort.mb = 100
> 12/06/13 12:03:30 INFO mapred.MapTask: data buffer = 79691776/99614720
> 12/06/13 12:03:30 INFO mapred.MapTask: record buffer = 262144/327680
> 12/06/13 12:03:30 INFO mapred.JobClient:  map 0% reduce 0%
> 12/06/13 12:03:35 INFO mapred.LocalJobRunner: Generated 95735000
> samples.
> 12/06/13 12:03:36 INFO mapred.JobClient:  map 100% reduce 0%
> 12/06/13 12:03:38 INFO mapred.LocalJobRunner: Generated 151872000
> samples.
>
> Here is the content of yarn-site.xml:
>
> <configuration>
>   <property>
>     <name>yarn.nodemanager.aux-services</name>
>     <value>mapreduce.shuffle</value>
>   </property>
>
>   <property>
>     <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>     <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>   </property>
>
>   <property>
>     <name>yarn.log-aggregation-enable</name>
>     <value>true</value>
>   </property>
>
>   <property>
>     <description>List of directories to store localized files in.</
> description>
>     <name>yarn.nodemanager.local-dirs</name>
>     <value>/disk/yarn/local</value>
>   </property>
>
>   <property>
>     <description>Where to store container logs.</description>
>     <name>yarn.nodemanager.log-dirs</name>
>     <value>/disk/yarn/logs</value>
>   </property>
>
>   <property>
>     <description>Where to aggregate logs to.</description>
>     <name>yarn.nodemanager.remote-app-log-dir</name>
>     <value>/var/log/hadoop-yarn/apps</value>
>   </property>
>
>   <property>
>     <description>Classpath for typical applications.</description>
>      <name>yarn.application.classpath</name>
>      <value>
>         $HADOOP_CONF_DIR,
Thanks & Regards,
Anil Gupta