Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Hadoop 2.2.0 :  Job not showing on resource management web UI


Copy link to this message
-
Hadoop 2.2.0 :  Job not showing on resource management web UI
Hi,
I have set up a single node Hadoop 2.2.0 .
I can submit  wordcount  hadoop job to YARN from the command line.
It run successfully and I checked the result is in hdfs.
matmsh@gauss:~> hadoop jar  ~/installed/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar  wordcount input output6
14/06/28 13:40:00 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/06/28 13:40:00 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/06/28 13:40:00 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/06/28 13:40:00 INFO input.FileInputFormat: Total input paths to process : 1
14/06/28 13:40:00 INFO mapreduce.JobSubmitter: number of splits:1
14/06/28 13:40:00 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/06/28 13:40:00 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/06/28 13:40:00 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/06/28 13:40:00 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local28288787_0001
14/06/28 13:40:00 WARN conf.Configuration: file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/06/28 13:40:00 WARN conf.Configuration: file:/tmp/hadoop-matmsh/mapred/staging/matmsh28288787/.staging/job_local28288787_0001/job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
14/06/28 13:40:00 WARN conf.Configuration: file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/06/28 13:40:00 WARN conf.Configuration: file:/tmp/hadoop-matmsh/mapred/local/localRunner/matmsh/job_local28288787_0001/job_local28288787_0001.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
14/06/28 13:40:00 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/06/28 13:40:00 INFO mapreduce.Job: Running job: job_local28288787_0001
14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/06/28 13:40:00 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/06/28 13:40:01 INFO mapred.LocalJobRunner: Waiting for map tasks
14/06/28 13:40:01 INFO mapred.LocalJobRunner: Starting task: attempt_local28288787_0001_m_000000_0
14/06/28 13:40:01 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
14/06/28 13:40:01 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/matmsh/input/getdoc.txt:0+23992
14/06/28 13:40:01 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/06/28 13:40:01 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
.....

However, the above job does not show up at  the web UI :
http://localhost:8088/cluster/apps

My configurations are below,
mapred-site.xml :
<configuration>

<property>
<value>mapreduce.framework.name</value>
<name>yarn</name>
</property>

</configuration>

hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/matmsh/temp/hadoop2/namenode</value>
<description>The name of the default file system.</description>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/matmsh/temp/hadoop2/datanode</value>
<description>The name of the default file system.</description>
</property>

</configuration>
yarn-site.xml
<configuration>

<!-- Site specific YARN configuration properties -->

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
What other configurations are  needed to show the job in the web UI ?
Thanks in advance for any  assistance !

Shing
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB