-Re: Cascading jobs in hadoop
bharath vissapragada 2009-10-03, 17:29
Tom and Chris ,
Thanks for your replies .. I have seen thr o.a.h.mapred.jobcontrol.Job
and o.a.h.mapreduce.Job .. Only one of them has the above option of
adding a dependent Jobs .. Can anyone tell me the difference between
"mapred" and "mapreduce" packages ..
Thanks in advance
On 10/2/09, Chris K Wensel <[EMAIL PROTECTED]> wrote:
> You might find the Cascading project quite useful in this regard.
> using MapReduceFlow and CascadeConnector classes, you can chain
> arbitrary MR jobs together. Cascading will determine the dependencies,
> if any, and run the jobs in topological order (independent jobs will
> be submitted to run in parallel).
> you may also find writing your own MR jobs by hand tedious and
> brittle. Cascading can help you there as well.
> On Oct 2, 2009, at 3:29 AM, bharath v wrote:
>> Hi all,
>> I have a set of map red jobs which need to be cascaded ,i.e, output
>> of MR
>> job1 is the input of MR job2. etc..
>> Can anyone point me to the corresponding classes in hadoop 0.20.0 API?
>> I have seen "x.addDependingJob(y)" function in the yahoo's hadoop
>> but that is for the older versions..
>> What is the similar thing in 0.20.0 API?
>> Any help is appreciated ,
>> IIIT Hyderabad!
> Chris K Wensel
> [EMAIL PROTECTED]