Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Yarn job runs in Local Mode even though the cluster is running in Distributed Mode


Copy link to this message
-
Re: Yarn job runs in Local Mode even though the cluster is running in Distributed Mode
Hi Marcus,

Sorry i forgot to mention that Job history server is installed and running
and AFAIK resourcemanager is responsible for running MR jobs. Historyserver
is only used to get info about MR jobs.

Thanks,
Anil

On Wed, Jun 13, 2012 at 2:04 PM, Marcos Ortiz <[EMAIL PROTECTED]> wrote:

> According to the CDH 4 official documentation, you should install a
> JobHistory server for your MRv2 (YARN)
> cluster.
> https://ccp.cloudera.com/**display/CDH4DOC/Deploying+**
> MapReduce+v2+%28YARN%29+on+a+**Cluster<https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster>
>
> How to configure the HistoryServer
> https://ccp.cloudera.com/**display/CDH4DOC/Deploying+**
> MapReduce+v2+%28YARN%29+on+a+**Cluster#DeployingMapReducev2%**
> 28YARN%29onaCluster-Step3<https://ccp.cloudera.com/display/CDH4DOC/Deploying+MapReduce+v2+%28YARN%29+on+a+Cluster#DeployingMapReducev2%28YARN%29onaCluster-Step3>
>
>
>
>
> On 06/13/2012 03:16 PM, anil gupta wrote:
>
>> Hi All
>>
>> I am using cdh4 for running a HBase cluster on CentOs6.0. I have 5
>> nodes in my cluster(2 Admin Node and 3 DN).
>> My resourcemanager is up and running and showing that all three DN are
>> running the nodemanager. HDFS is also working fine and showing 3 DN's.
>>
>> But when i fire the pi example job. It starts to run in Local mode.
>> Here is the console output:
>> sudo -u hdfs yarn jar /usr/lib/hadoop-mapreduce/**hadoop-mapreduce-
>> examples.jar pi 10 1000000000
>> Number of Maps  = 10
>> Samples per Map = 1000000000
>> Wrote input for Map #0
>> Wrote input for Map #1
>> Wrote input for Map #2
>> Wrote input for Map #3
>> Wrote input for Map #4
>> Wrote input for Map #5
>> Wrote input for Map #6
>> Wrote input for Map #7
>> Wrote input for Map #8
>> Wrote input for Map #9
>> Starting Job
>> 12/06/13 12:03:27 WARN conf.Configuration: session.id is deprecated.
>> Instead, use dfs.metrics.session-id
>> 12/06/13 12:03:27 INFO jvm.JvmMetrics: Initializing JVM Metrics with
>> processName=JobTracker, sessionId>> 12/06/13 12:03:27 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 12/06/13 12:03:27 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the
>> same.
>> 12/06/13 12:03:28 INFO mapred.FileInputFormat: Total input paths to
>> process : 10
>> 12/06/13 12:03:29 INFO mapred.JobClient: Running job: job_local_0001
>> 12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter set in
>> config null
>> 12/06/13 12:03:29 INFO mapred.LocalJobRunner: OutputCommitter is
>> org.apache.hadoop.mapred.**FileOutputCommitter
>> 12/06/13 12:03:29 WARN mapreduce.Counters: Group
>> org.apache.hadoop.mapred.Task$**Counter is deprecated. Use
>> org.apache.hadoop.mapreduce.**TaskCounter instead
>> 12/06/13 12:03:29 INFO util.ProcessTree: setsid exited with exit code
>> 0
>> 12/06/13 12:03:29 INFO mapred.Task:  Using ResourceCalculatorPlugin :
>> org.apache.hadoop.util.**LinuxResourceCalculatorPlugin@**3d46e381
>> 12/06/13 12:03:29 WARN mapreduce.Counters: Counter name
>> MAP_INPUT_BYTES is deprecated. Use FileInputFormatCounters as group
>> name and  BYTES_READ as counter name instead
>> 12/06/13 12:03:29 INFO mapred.MapTask: numReduceTasks: 1
>> 12/06/13 12:03:29 INFO mapred.MapTask: io.sort.mb = 100
>> 12/06/13 12:03:30 INFO mapred.MapTask: data buffer = 79691776/99614720
>> 12/06/13 12:03:30 INFO mapred.MapTask: record buffer = 262144/327680
>> 12/06/13 12:03:30 INFO mapred.JobClient:  map 0% reduce 0%
>> 12/06/13 12:03:35 INFO mapred.LocalJobRunner: Generated 95735000
>> samples.
>> 12/06/13 12:03:36 INFO mapred.JobClient:  map 100% reduce 0%
>> 12/06/13 12:03:38 INFO mapred.LocalJobRunner: Generated 151872000
>> samples.
>>
>> Here is the content of yarn-site.xml:
>>
>> <configuration>
>>   <property>
>>     <name>yarn.nodemanager.aux-**services</name>
>>     <value>mapreduce.shuffle</**value>
>>   </property>
>>
>>   <property>
>>     <name>yarn.nodemanager.aux-**services.mapreduce.shuffle.**

Thanks & Regards,
Anil Gupta
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB