Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Submitting MapReduce job from remote server using JobClient


+
Amit Sela 2013-01-24, 12:13
+
Harsh J 2013-01-24, 15:12
+
Amit Sela 2013-01-24, 16:15
+
Amit Sela 2013-01-27, 11:43
Copy link to this message
-
Re: Submitting MapReduce job from remote server using JobClient
Hello Amit,

I tried the same scenario, submitting map reduce jobs from a system that is
outside the hadoop cluster and I used Sring Hadoop to do it. It worked
wonderfully. Spring has made alot of things easier...
you can try it. Here is a reference on how to do it:

http://www.petrikainulainen.net/programming/apache-hadoop/creating-hadoop-mapreduce-job-with-spring-data-apache-hadoop/

hope this helps,
Regards,

On Sun, Jan 27, 2013 at 12:43 PM, Amit Sela <[EMAIL PROTECTED]> wrote:

> Yes I do.
> I checked that by printing out Configuration.toString() and I see only the
> files I add as resources.
> Moreover, in my test environment, the test Analytics server is also a data
> node (or maybe that could cause more trouble ?).
> Anyway, I still get
> *org.apache.hadoop.mapred.JobClient                           - Running
> job: job_local_0001*
> *
> *
> And I don't know what's wrong here, I create a new Configuration(false) to
> avoid default settings. I set the resources manually (addResource). I
> validate it. Anything I'm forgetting ?
>
>
> On Thu, Jan 24, 2013 at 9:49 PM, <[EMAIL PROTECTED]> wrote:
>
>> **
>> Hi Amit,
>>
>> Apart for the hadoop jars, Do you have the same config files
>> ($HADOOP_HOME/conf) that are in the cluster on your analytics server as
>> well?
>>
>> If you are having the default config files in analytics server then your
>> MR job would be running locally and not on the cluster.
>> Regards
>> Bejoy KS
>>
>> Sent from remote device, Please excuse typos
>> ------------------------------
>> *From: * Amit Sela <[EMAIL PROTECTED]>
>> *Date: *Thu, 24 Jan 2013 18:15:49 +0200
>> *To: *<[EMAIL PROTECTED]>
>> *ReplyTo: * [EMAIL PROTECTED]
>> *Subject: *Re: Submitting MapReduce job from remote server using
>> JobClient
>>
>> Hi Harsh,
>> I'm using Job.waitForCompletion() method to run the job but I can't see
>> it in the webapp and it doesn't seem to finish...
>> I get:
>>  *org.apache.hadoop.mapred.JobClient                           - Running
>> job: job_local_0001*
>> *INFO  org.apache.hadoop.util.ProcessTree                           -
>> setsid exited with exit code 0*
>> *2013-01-24 08:10:12.521 [Thread-51]                INFO
>>  org.apache.hadoop.mapred.Task                                -  Using
>> ResourceCalculatorPlugin :
>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@7db1be6*
>> *2013-01-24 08:10:12.536 [Thread-51]                INFO
>>  org.apache.hadoop.mapred.MapTask                             - io.sort.mb
>> = 100*
>> *2013-01-24 08:10:12.573 [Thread-51]                INFO
>>  org.apache.hadoop.mapred.MapTask                             - data buffer
>> = 79691776/99614720*
>> *2013-01-24 08:10:12.573 [Thread-51]                INFO
>>  org.apache.hadoop.mapred.MapTask                             - record
>> buffer = 262144/327680*
>> *2013-01-24 08:10:12.599 [Thread-51]                INFO
>>  org.apache.hadoop.mapred.MapTask                             - Starting
>> flush of map output*
>> *2013-01-24 08:10:12.608 [Thread-51]                INFO
>>  org.apache.hadoop.mapred.Task                                -
>> Task:attempt_local_0001_m_000000_0 is done. And is in the process of
>> commiting*
>> *2013-01-24 08:10:13.348
>> [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1]
>> INFO  org.apache.hadoop.mapred.JobClient                           -  map
>> 0% reduce 0%*
>> *2013-01-24 08:10:15.509 [Thread-51]                INFO
>>  org.apache.hadoop.mapred.LocalJobRunner                      - *
>> *2013-01-24 08:10:15.510 [Thread-51]                INFO
>>  org.apache.hadoop.mapred.Task                                - Task
>> 'attempt_local_0001_m_000000_0' done.*
>> *2013-01-24 08:10:15.511 [Thread-51]                INFO
>>  org.apache.hadoop.mapred.Task                                -  Using
>> ResourceCalculatorPlugin :
>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@6b02b23d*
>> *2013-01-24 08:10:15.512 [Thread-51]                INFO
Regards,
Ouch Whisper
010101010101
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB