Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - Yarn job runs in Local Mode even though the cluster is running in Distributed Mode


+
anil gupta 2012-06-13, 19:16
+
anil gupta 2012-06-13, 19:27
+
Marcos Ortiz 2012-06-13, 21:04
+
anil gupta 2012-06-13, 21:26
+
Harsh J 2012-06-14, 05:41
Copy link to this message
-
Re: Yarn job runs in Local Mode even though the cluster is running in Distributed Mode
Marcos Ortiz 2012-06-14, 02:34
Can you share with us in pastebin all conf files that you are using for
YARN?
On 06/13/2012 05:26 PM, anil gupta wrote:
> Hi Marcus,
>
> Sorry i forgot to mention that Job history server is installed and
> running and AFAIK resourcemanager is responsible for running MR jobs.
> Historyserver is only used to get info about MR jobs.
>
> Thanks,
> Anil
>
> On Wed, Jun 13, 2012 at 2:04 PM, Marcos Ortiz <[EMAIL PROTECTED]
> <mailto:[EMAIL PROTECTED]>> wrote:
>
>     According to the CDH 4 official documentation, you should install
>     a JobHistory server for your MRv2 (YARN)
>     cluster.
>     https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster
>
>     How to configure the HistoryServer
>     https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster#DeployingMapReducev2%28YARN%29onaCluster-Step3
>
>
>
>
>
>     On 06/13/2012 03:16 PM, anil gupta wrote:
>
>         Hi All
>
>         I am using cdh4 for running a HBase cluster on CentOs6.0. I have 5
>         nodes in my cluster(2 Admin Node and 3 DN).
>         My resourcemanager is up and running and showing that all
>         three DN are
>         running the nodemanager. HDFS is also working fine and showing
>         3 DN's.
>
>         But when i fire the pi example job. It starts to run in Local
>         mode.
>         Here is the console output:
>         sudo -u hdfs yarn jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-
>         examples.jar pi 10 1000000000
>         Number of Maps  = 10
>         Samples per Map = 1000000000
>         Wrote input for Map #0
>         Wrote input for Map #1
>         Wrote input for Map #2
>         Wrote input for Map #3
>         Wrote input for Map #4
>         Wrote input for Map #5
>         Wrote input for Map #6
>         Wrote input for Map #7
>         Wrote input for Map #8
>         Wrote input for Map #9
>         Starting Job
>         12/06/13 12:03:27 WARN conf.Configuration: session.id
>         <http://session.id> is deprecated.
>         Instead, use dfs.metrics.session-id
>         12/06/13 12:03:27 INFO jvm.JvmMetrics: Initializing JVM
>         Metrics with
>         processName=JobTracker, sessionId>         12/06/13 12:03:27 INFO util.NativeCodeLoader: Loaded the
>         native-hadoop
>         library
>         12/06/13 12:03:27 WARN mapred.JobClient: Use
>         GenericOptionsParser for
>         parsing the arguments. Applications should implement Tool for the
>         same.
>         12/06/13 12:03:28 INFO mapred.FileInputFormat: Total input
>         paths to
>         process : 10
>         12/06/13 12:03:29 INFO mapred.JobClient: Running job:
>         job_local_0001
>         12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter
>         set in
>         config null
>         12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter is
>         org.apache.hadoop.mapred.FileOutputCommitter
>         12/06/13 12:03:29 WARN mapreduce.Counters: Group
>         org.apache.hadoop.mapred.Task$Counter is deprecated. Use
>         org.apache.hadoop.mapreduce.TaskCounter instead
>         12/06/13 12:03:29 INFO util.ProcessTree: setsid exited with
>         exit code
>         0
>         12/06/13 12:03:29 INFO mapred.Task:  Using
>         ResourceCalculatorPlugin :
>         org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3d46e381
>         12/06/13 12:03:29 WARN mapreduce.Counters: Counter name
>         MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as
>         group
>         name and  BYTES_READ as counter name instead
>         12/06/13 12:03:29 INFO mapred.MapTask: numReduceTasks: 1
>         12/06/13 12:03:29 INFO mapred.MapTask: io.sort.mb = 100
>         12/06/13 12:03:30 INFO mapred.MapTask: data buffer >         79691776/99614720
>         12/06/13 12:03:30 INFO mapred.MapTask: record buffer >         262144/327680
>         12/06/13 12:03:30 INFO mapred.JobClient:  map 0% reduce 0%
>         12/06/13 12:03:35 INFO mapred.LocalJobRunner: Generated 95735000

Marcos Luis Ort�z Valmaseda
  Data Engineer&&  Sr. System Administrator at UCI
  http://marcosluis2186.posterous.com
  http://www.linkedin.com/in/marcosluis2186
  Twitter: @marcosluis2186

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci