Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> How to automate running pig scripts

Juan Martin Pampliega 2012-05-26, 18:09
Copy link to this message
Re: How to automate running pig scripts
Take a look at Oozie or Azkaban.


I prefer Oozie, its a lot more comprehensive. AZK does not seem to be fully
baked (though installation and getting started could be easier).

On Sat, May 26, 2012 at 11:09 AM, Juan Martin Pampliega <

> Hi,
> I need to have pig scripts run automatically every certain amount of time.
> Also, I need to know if a script is still running so that the next one does
> not start until the previous one has finished and also I need to register
> somewhere the end state of each script run to display it in a table on a
> web page.
> My last requirement is to have the scripts receive some parameters that
> will be from which id to start processing. So for example the second time
> the script is run it should know which was the last id that was processed
> and start from there onwards.
> Does anyone have any suggestions on how to approach this?
> I was thinking of using Jython to run the Pig scripts programmatically but
> I'm having trouble calling all these from Java and I'm always getting
> success status even when the pig script fails.
> Any other suggestions?
Juan Martin Pampliega 2012-05-26, 19:23