Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Submitting MapReduce job from remote server using JobClient


+
Amit Sela 2013-01-24, 12:13
+
Harsh J 2013-01-24, 15:12
Copy link to this message
-
Re: Submitting MapReduce job from remote server using JobClient
Hi Harsh,
I'm using Job.waitForCompletion() method to run the job but I can't see it
in the webapp and it doesn't seem to finish...
I get:
 *org.apache.hadoop.mapred.JobClient                           - Running
job: job_local_0001*
*INFO  org.apache.hadoop.util.ProcessTree                           -
setsid exited with exit code 0*
*2013-01-24 08:10:12.521 [Thread-51]                INFO
 org.apache.hadoop.mapred.Task                                -  Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7db1be6*
*2013-01-24 08:10:12.536 [Thread-51]                INFO
 org.apache.hadoop.mapred.MapTask                             - io.sort.mb
= 100*
*2013-01-24 08:10:12.573 [Thread-51]                INFO
 org.apache.hadoop.mapred.MapTask                             - data buffer
= 79691776/99614720*
*2013-01-24 08:10:12.573 [Thread-51]                INFO
 org.apache.hadoop.mapred.MapTask                             - record
buffer = 262144/327680*
*2013-01-24 08:10:12.599 [Thread-51]                INFO
 org.apache.hadoop.mapred.MapTask                             - Starting
flush of map output*
*2013-01-24 08:10:12.608 [Thread-51]                INFO
 org.apache.hadoop.mapred.Task                                -
Task:attempt_local_0001_m_000000_0 is done. And is in the process of
commiting*
*2013-01-24 08:10:13.348
[org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1]
INFO  org.apache.hadoop.mapred.JobClient                           -  map
0% reduce 0%*
*2013-01-24 08:10:15.509 [Thread-51]                INFO
 org.apache.hadoop.mapred.LocalJobRunner                      - *
*2013-01-24 08:10:15.510 [Thread-51]                INFO
 org.apache.hadoop.mapred.Task                                - Task
'attempt_local_0001_m_000000_0' done.*
*2013-01-24 08:10:15.511 [Thread-51]                INFO
 org.apache.hadoop.mapred.Task                                -  Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@6b02b23d*
*2013-01-24 08:10:15.512 [Thread-51]                INFO
 org.apache.hadoop.mapred.MapTask                             - io.sort.mb
= 100*
*2013-01-24 08:10:15.549 [Thread-51]                INFO
 org.apache.hadoop.mapred.MapTask                             - data buffer
= 79691776/99614720*
*2013-01-24 08:10:15.550 [Thread-51]                INFO
 org.apache.hadoop.mapred.MapTask                             - record
buffer = 262144/327680*
*2013-01-24 08:10:15.557 [Thread-51]                INFO
 org.apache.hadoop.mapred.MapTask                             - Starting
flush of map output*
*2013-01-24 08:10:15.560 [Thread-51]                INFO
 org.apache.hadoop.mapred.Task                                -
Task:attempt_local_0001_m_000001_0 is done. And is in the process of
commiting*
*2013-01-24 08:10:16.358
[org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1]
INFO  org.apache.hadoop.mapred.JobClient                           -  map
100% reduce 0%*

And after that, instead of going to Reduce phase I keep getting map
attempts like:

*INFO  org.apache.hadoop.mapred.MapTask                             -
io.sort.mb = 100*
*2013-01-24 08:10:21.563 [Thread-51]                INFO
 org.apache.hadoop.mapred.MapTask                             - data buffer
= 79691776/99614720*
*2013-01-24 08:10:21.563 [Thread-51]                INFO
 org.apache.hadoop.mapred.MapTask                             - record
buffer = 262144/327680*
*2013-01-24 08:10:21.570 [Thread-51]                INFO
 org.apache.hadoop.mapred.MapTask                             - Starting
flush of map output*
*2013-01-24 08:10:21.573 [Thread-51]                INFO
 org.apache.hadoop.mapred.Task                                -
Task:attempt_local_0001_m_000003_0 is done. And is in the process of
commiting*
*2013-01-24 08:10:24.529 [Thread-51]                INFO
 org.apache.hadoop.mapred.LocalJobRunner                      - *
*2013-01-24 08:10:24.529 [Thread-51]                INFO
 org.apache.hadoop.mapred.Task                                - Task
'attempt_local_0001_m_000003_0' done.*
*2013-01-24 08:10:24.530 [Thread-51]                INFO
 org.apache.hadoop.mapred.Task                                -  Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@42e87d99*
*
*
Any clues ?
Thanks for the help.

On Thu, Jan 24, 2013 at 5:12 PM, Harsh J <[EMAIL PROTECTED]> wrote:

+
Amit Sela 2013-01-27, 11:43
+
Panshul Whisper 2013-01-27, 11:53
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB