Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Submitting MapReduce job from remote server using JobClient


+
Amit Sela 2013-01-24, 12:13
+
Harsh J 2013-01-24, 15:12
+
Amit Sela 2013-01-24, 16:15
Copy link to this message
-
Re: Submitting MapReduce job from remote server using JobClient
Yes I do.
I checked that by printing out Configuration.toString() and I see only the
files I add as resources.
Moreover, in my test environment, the test Analytics server is also a data
node (or maybe that could cause more trouble ?).
Anyway, I still get
*org.apache.hadoop.mapred.JobClient                           - Running
job: job_local_0001*
*
*
And I don't know what's wrong here, I create a new Configuration(false) to
avoid default settings. I set the resources manually (addResource). I
validate it. Anything I'm forgetting ?
On Thu, Jan 24, 2013 at 9:49 PM, <[EMAIL PROTECTED]> wrote:

> **
> Hi Amit,
>
> Apart for the hadoop jars, Do you have the same config files
> ($HADOOP_HOME/conf) that are in the cluster on your analytics server as
> well?
>
> If you are having the default config files in analytics server then your
> MR job would be running locally and not on the cluster.
> Regards
> Bejoy KS
>
> Sent from remote device, Please excuse typos
> ------------------------------
> *From: * Amit Sela <[EMAIL PROTECTED]>
> *Date: *Thu, 24 Jan 2013 18:15:49 +0200
> *To: *<[EMAIL PROTECTED]>
> *ReplyTo: * [EMAIL PROTECTED]
> *Subject: *Re: Submitting MapReduce job from remote server using JobClient
>
> Hi Harsh,
> I'm using Job.waitForCompletion() method to run the job but I can't see it
> in the webapp and it doesn't seem to finish...
> I get:
>  *org.apache.hadoop.mapred.JobClient                           - Running
> job: job_local_0001*
> *INFO  org.apache.hadoop.util.ProcessTree                           -
> setsid exited with exit code 0*
> *2013-01-24 08:10:12.521 [Thread-51]                INFO
>  org.apache.hadoop.mapred.Task                                -  Using
> ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7db1be6*
> *2013-01-24 08:10:12.536 [Thread-51]                INFO
>  org.apache.hadoop.mapred.MapTask                             - io.sort.mb
> = 100*
> *2013-01-24 08:10:12.573 [Thread-51]                INFO
>  org.apache.hadoop.mapred.MapTask                             - data buffer
> = 79691776/99614720*
> *2013-01-24 08:10:12.573 [Thread-51]                INFO
>  org.apache.hadoop.mapred.MapTask                             - record
> buffer = 262144/327680*
> *2013-01-24 08:10:12.599 [Thread-51]                INFO
>  org.apache.hadoop.mapred.MapTask                             - Starting
> flush of map output*
> *2013-01-24 08:10:12.608 [Thread-51]                INFO
>  org.apache.hadoop.mapred.Task                                -
> Task:attempt_local_0001_m_000000_0 is done. And is in the process of
> commiting*
> *2013-01-24 08:10:13.348
> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1]
> INFO  org.apache.hadoop.mapred.JobClient                           -  map
> 0% reduce 0%*
> *2013-01-24 08:10:15.509 [Thread-51]                INFO
>  org.apache.hadoop.mapred.LocalJobRunner                      - *
> *2013-01-24 08:10:15.510 [Thread-51]                INFO
>  org.apache.hadoop.mapred.Task                                - Task
> 'attempt_local_0001_m_000000_0' done.*
> *2013-01-24 08:10:15.511 [Thread-51]                INFO
>  org.apache.hadoop.mapred.Task                                -  Using
> ResourceCalculatorPlugin :
> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@6b02b23d*
> *2013-01-24 08:10:15.512 [Thread-51]                INFO
>  org.apache.hadoop.mapred.MapTask                             - io.sort.mb
> = 100*
> *2013-01-24 08:10:15.549 [Thread-51]                INFO
>  org.apache.hadoop.mapred.MapTask                             - data buffer
> = 79691776/99614720*
> *2013-01-24 08:10:15.550 [Thread-51]                INFO
>  org.apache.hadoop.mapred.MapTask                             - record
> buffer = 262144/327680*
> *2013-01-24 08:10:15.557 [Thread-51]                INFO
>  org.apache.hadoop.mapred.MapTask                             - Starting
> flush of map output*
> *2013-01-24 08:10
+
Panshul Whisper 2013-01-27, 11:53
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB