Hi,

I need to have pig scripts run automatically every certain amount of time.
Also, I need to know if a script is still running so that the next one does
not start until the previous one has finished and also I need to register
somewhere the end state of each script run to display it in a table on a
web page.
My last requirement is to have the scripts receive some parameters that
will be from which id to start processing. So for example the second time
the script is run it should know which was the last id that was processed
and start from there onwards.

Does anyone have any suggestions on how to approach this?

I was thinking of using Jython to run the Pig scripts programmatically but
I'm having trouble calling all these from Java and I'm always getting
success status even when the pig script fails.

Any other suggestions?

Take a look at Oozie or Azkaban.

http://incubator.apache.org/oozie/

I prefer Oozie, its a lot more comprehensive. AZK does not seem to be fully
baked (though installation and getting started could be easier).

On Sat, May 26, 2012 at 11:09 AM, Juan Martin Pampliega <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I need to have pig scripts run automatically every certain amount of time.
> Also, I need to know if a script is still running so that the next one does
> not start until the previous one has finished and also I need to register
> somewhere the end state of each script run to display it in a table on a
> web page.
> My last requirement is to have the scripts receive some parameters that
> will be from which id to start processing. So for example the second time
> the script is run it should know which was the last id that was processed
> and start from there onwards.
>
> Does anyone have any suggestions on how to approach this?
>
> I was thinking of using Jython to run the Pig scripts programmatically but
> I'm having trouble calling all these from Java and I'm always getting
> success status even when the pig script fails.
>
> Any other suggestions?
>

Thanks a lot Prashant. This seems exactly what I was looking for. Just one
doubt I have and it doesn't seem to be clear from the documentation. Do
this workflow managers work with pig jobs running in local mode or only in
MR mode?

On Sat, May 26, 2012 at 3:19 PM, Prashant Kommireddi <[EMAIL PROTECTED]>wrote:

> Take a look at Oozie or Azkaban.
>
> http://incubator.apache.org/oozie/
>
> I prefer Oozie, its a lot more comprehensive. AZK does not seem to be fully
> baked (though installation and getting started could be easier).
>
> On Sat, May 26, 2012 at 11:09 AM, Juan Martin Pampliega <
> [EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > I need to have pig scripts run automatically every certain amount of
> time.
> > Also, I need to know if a script is still running so that the next one
> does
> > not start until the previous one has finished and also I need to register
> > somewhere the end state of each script run to display it in a table on a
> > web page.
> > My last requirement is to have the scripts receive some parameters that
> > will be from which id to start processing. So for example the second time
> > the script is run it should know which was the last id that was processed
> > and start from there onwards.
> >
> > Does anyone have any suggestions on how to approach this?
> >
> > I was thinking of using Jython to run the Pig scripts programmatically
> but
> > I'm having trouble calling all these from Java and I'm always getting
> > success status even when the pig script fails.
> >
> > Any other suggestions?
> >
>