Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Cascading jobs in hadoop


Copy link to this message
-
Re: Cascading jobs in hadoop
Tom and Chris ,

Thanks for your replies .. I have seen thr o.a.h.mapred.jobcontrol.Job
and o.a.h.mapreduce.Job .. Only one of them has the above option of
adding a dependent Jobs .. Can anyone tell me the difference between
"mapred" and "mapreduce" packages ..

Thanks in advance

On 10/2/09, Chris K Wensel <[EMAIL PROTECTED]> wrote:
> You might find the Cascading project quite useful in this regard.
>
> http://www.cascading.org/
>
> using MapReduceFlow and CascadeConnector classes, you can chain
> arbitrary MR jobs together. Cascading will determine the dependencies,
> if any, and run the jobs in topological order (independent jobs will
> be submitted to run in parallel).
>
> you may also find writing your own MR jobs by hand tedious and
> brittle. Cascading can help you there as well.
>
> cheers,
> chris
>
> On Oct 2, 2009, at 3:29 AM, bharath v wrote:
>
>> Hi all,
>>
>> I have a set of map red jobs which need to be cascaded ,i.e, output
>> of MR
>> job1 is the input of MR job2. etc..
>>
>> Can anyone point me to the corresponding classes in hadoop 0.20.0 API?
>>
>> I have seen "x.addDependingJob(y)" function in the yahoo's hadoop
>> tutorial
>> but that is for the older versions..
>> What is the similar thing in 0.20.0 API?
>>
>> Any help is appreciated ,
>>
>> Thanks
>> bharath.v
>> ug3
>> IIIT Hyderabad!
>
> --
> Chris K Wensel
> [EMAIL PROTECTED]
> http://www.concurrentinc.com
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB