Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Resume failed pig script


Copy link to this message
-
Resume failed pig script
In production I use short Pig scripts and schedule them with Azkaban
with dependencies setup, so that I can use Azkaban to restart long
data pipelines at the point of failure. I edit the failing pig script,
usually towards the end of the data pipeline, and restart the Azkaban
job. This saves hours and hours of repeated processing.

I wish Pig could do this. To resume at its point of failure when
re-run from the command line. Is this feasible?

Russell Jurney
twitter.com/rjurney
[EMAIL PROTECTED]
datasyndrome.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB