Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> Yarn job runs in Local Mode even though the cluster is running in Distributed Mode


+
anil gupta 2012-06-13, 19:16
+
anil gupta 2012-06-13, 19:27
+
Marcos Ortiz 2012-06-13, 21:04
+
anil gupta 2012-06-13, 21:26
+
Harsh J 2012-06-14, 05:41
Copy link to this message
-
Re: Yarn job runs in Local Mode even though the cluster is running in Distributed Mode
Can you share with us in pastebin all conf files that you are using for
YARN?
On 06/13/2012 05:26 PM, anil gupta wrote:
> Hi Marcus,
>
> Sorry i forgot to mention that Job history server is installed and
> running and AFAIK resourcemanager is responsible for running MR jobs.
> Historyserver is only used to get info about MR jobs.
>
> Thanks,
> Anil
>
> On Wed, Jun 13, 2012 at 2:04 PM, Marcos Ortiz <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     According to the CDH 4 official documentation, you should install
>     a JobHistory server for your MRv2 (YARN)
>     cluster.
>     https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster
>
>     How to configure the HistoryServer
>     https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster#DeployingMapReducev2%28YARN%29onaCluster-Step3
>
>
>
>
>
>     On 06/13/2012 03:16 PM, anil gupta wrote:
>
>         Hi All
>
>         I am using cdh4 for running a HBase cluster on CentOs6.0. I have 5
>         nodes in my cluster(2 Admin Node and 3 DN).
>         My resourcemanager is up and running and showing that all
>         three DN are
>         running the nodemanager. HDFS is also working fine and showing
>         3 DN's.
>
>         But when i fire the pi example job. It starts to run in Local
>         mode.
>         Here is the console output:
>         sudo -u hdfs yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-
>         examples.jar pi 10 1000000000
>         Number of Maps  = 10
>         Samples per Map = 1000000000
>         Wrote input for Map #0
>         Wrote input for Map #1
>         Wrote input for Map #2
>         Wrote input for Map #3
>         Wrote input for Map #4
>         Wrote input for Map #5
>         Wrote input for Map #6
>         Wrote input for Map #7
>         Wrote input for Map #8
>         Wrote input for Map #9
>         Starting Job
>         12/06/13 12:03:27 WARN conf.Configuration: session.id
>         <http://session.id> is deprecated.
>         Instead, use dfs.metrics.session-id
>         12/06/13 12:03:27 INFO jvm.JvmMetrics: Initializing JVM
>         Metrics with
>         processName=JobTracker, sessionId>         12/06/13 12:03:27 INFO util.NativeCodeLoader: Loaded the
>         native-hadoop
>         library
>         12/06/13 12:03:27 WARN mapred.JobClient: Use
>         GenericOptionsParser for
>         parsing the arguments. Applications should implement Tool for the
>         same.
>         12/06/13 12:03:28 INFO mapred.FileInputFormat: Total input
>         paths to
>         process : 10
>         12/06/13 12:03:29 INFO mapred.JobClient: Running job:
>         job_local_0001
>         12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter
>         set in
>         config null
>         12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter is
>         org.apache.hadoop.mapred.FileOutputCommitter
>         12/06/13 12:03:29 WARN mapreduce.Counters: Group
>         org.apache.hadoop.mapred.Task$Counter is deprecated. Use
>         org.apache.hadoop.mapreduce.TaskCounter instead
>         12/06/13 12:03:29 INFO util.ProcessTree: setsid exited with
>         exit code
>         0
>         12/06/13 12:03:29 INFO mapred.Task:  Using
>         ResourceCalculatorPlugin :
>         org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3d46e381
>         12/06/13 12:03:29 WARN mapreduce.Counters: Counter name
>         MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as
>         group
>         name and  BYTES_READ as counter name instead
>         12/06/13 12:03:29 INFO mapred.MapTask: numReduceTasks: 1
>         12/06/13 12:03:29 INFO mapred.MapTask: io.sort.mb = 100
>         12/06/13 12:03:30 INFO mapred.MapTask: data buffer >         79691776/99614720
>         12/06/13 12:03:30 INFO mapred.MapTask: record buffer >         262144/327680
>         12/06/13 12:03:30 INFO mapred.JobClient:  map 0% reduce 0%
>         12/06/13 12:03:35 INFO mapred.LocalJobRunner: Generated 95735000

Marcos Luis Ort�z Valmaseda
  Data Engineer&&  Sr. System Administrator at UCI
  http://marcosluis2186.posterous.com
  http://www.linkedin.com/in/marcosluis2186
  Twitter: @marcosluis2186

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci