Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig, mail # user - Managing Large Pig Scripts


+
Christopher Petrino 2014-03-05, 15:33
+
Dan DeCapria, CivicScien.... 2014-03-05, 15:55
Copy link to this message
-
Re: Managing Large Pig Scripts
Jacob Perkins 2014-03-05, 15:56
Christopher,

You might consider breaking it into one or more reusable macros. What version of pig are you using?

For complicated scripts, especially if you didn't write them, you might want to take a look at lipstick, https://github.com/Netflix/Lipstick
It allows you to visualize the dag and clearly shows what logical operators map to map-reduce jobs. It could be a starting point for managing complexity at least.

@thedatachef
On Mar 5, 2014, at 7:33 AM, Christopher Petrino <[EMAIL PROTECTED]> wrote:

 
+
Christopher Petrino 2014-03-05, 18:02
+
Russell Jurney 2014-03-05, 18:12