Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> How to automate running pig scripts


+
Juan Martin Pampliega 2012-05-26, 18:09
Copy link to this message
-
Re: How to automate running pig scripts
Take a look at Oozie or Azkaban.

http://incubator.apache.org/oozie/

I prefer Oozie, its a lot more comprehensive. AZK does not seem to be fully
baked (though installation and getting started could be easier).

On Sat, May 26, 2012 at 11:09 AM, Juan Martin Pampliega <
[EMAIL PROTECTED]> wrote:

> Hi,
>
> I need to have pig scripts run automatically every certain amount of time.
> Also, I need to know if a script is still running so that the next one does
> not start until the previous one has finished and also I need to register
> somewhere the end state of each script run to display it in a table on a
> web page.
> My last requirement is to have the scripts receive some parameters that
> will be from which id to start processing. So for example the second time
> the script is run it should know which was the last id that was processed
> and start from there onwards.
>
> Does anyone have any suggestions on how to approach this?
>
> I was thinking of using Jython to run the Pig scripts programmatically but
> I'm having trouble calling all these from Java and I'm always getting
> success status even when the pig script fails.
>
> Any other suggestions?
>
+
Juan Martin Pampliega 2012-05-26, 19:23
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB