Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Fwd: How to automate running pig scripts


Copy link to this message
-
Fwd: How to automate running pig scripts
Forwarding your question to the Oozie mailing list.  cc Pig users

Does Oozie work with pig jobs running in local mode or only in MR mode?

Thanks,

---------- Forwarded message ----------
From: Juan Martin Pampliega <[EMAIL PROTECTED]>
Date: Sat, May 26, 2012 at 12:23 PM
Subject: Re: How to automate running pig scripts
To: [EMAIL PROTECTED]
Thanks a lot Prashant. This seems exactly what I was looking for. Just one
doubt I have and it doesn't seem to be clear from the documentation. Do
this workflow managers work with pig jobs running in local mode or only in
MR mode?

On Sat, May 26, 2012 at 3:19 PM, Prashant Kommireddi <[EMAIL PROTECTED]
>wrote:

> Take a look at Oozie or Azkaban.
>
> http://incubator.apache.org/oozie/
>
> I prefer Oozie, its a lot more comprehensive. AZK does not seem to be
fully
> baked (though installation and getting started could be easier).
>
> On Sat, May 26, 2012 at 11:09 AM, Juan Martin Pampliega <
> [EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > I need to have pig scripts run automatically every certain amount of
> time.
> > Also, I need to know if a script is still running so that the next one
> does
> > not start until the previous one has finished and also I need to
register
> > somewhere the end state of each script run to display it in a table on a
> > web page.
> > My last requirement is to have the scripts receive some parameters that
> > will be from which id to start processing. So for example the second
time
> > the script is run it should know which was the last id that was
processed
> > and start from there onwards.
> >
> > Does anyone have any suggestions on how to approach this?
> >
> > I was thinking of using Jython to run the Pig scripts programmatically
> but
> > I'm having trouble calling all these from Java and I'm always getting
> > success status even when the pig script fails.
> >
> > Any other suggestions?
> >
>