Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # dev - Persisting Pig Scripts


+
Prashant Kommireddi 2012-06-06, 22:52
Copy link to this message
-
Re: Persisting Pig Scripts
Daniel Dai 2012-06-06, 23:19
One existing solution is "pig.script" entry inside job.xml, it is the
serialized Pig script. JobHistoryLoader can load job.xml files and grab
those entries. Does that solve your problem?

Daniel

On Wed, Jun 6, 2012 at 3:52 PM, Prashant Kommireddi <[EMAIL PROTECTED]>wrote:

> Hi All,
>
> What do you guys think about adding a feature to be able to persist the
> script (file or cache in case of grunt) on HDFS or locally based on an
> admin setting (pig.properties). This will help infrastructure/ops teams
> analyze nature of Pig scripts and be able to make certain decisions based
> on it (optimizing data storage based on access patterns etc). This is
> actually something we want to do but the challenge is there is no central
> place where we can track user scripts.
>
> It could be a config param "pig.persist.script=/pig/". The script could be
> stored with a configurable name -> ${mapred.job.name}+${user.name
> }+timestamp"
> either on HDFS or local based on the configuration setting.
>
> Thanks,
> Prashant
>
+
Prashant Kommireddi 2012-06-07, 00:16
+
Dmitriy Ryaboy 2012-06-07, 00:39
+
Prashant Kommireddi 2012-06-07, 00:44
+
Bill Graham 2012-06-07, 00:56
+
Prashant Kommireddi 2012-06-11, 21:33
+
Bill Graham 2012-06-11, 22:01
+
Jonathan Coveney 2012-06-11, 22:17