-Re: Pig and Storm
Pradeep Gollakota 2013-07-25, 03:43
I've added a wiki page for a "Pig on Storm Proposal" at
I've included a primer on Storm (and Trident) as well as some of the
challenges I foresee. Please read though my proposal and let me know what
your thoughts are.
On Wed, Jul 24, 2013 at 2:36 PM, Alan Gates <[EMAIL PROTECTED]> wrote:
> This sounds exciting. The next question is how do you plan to do it?
> Would a physical plan be translated to a Storm job (or jobs)? Would it
> need a different physical plan? Or would you just have the connection at
> the language layer and all the planning separate? Do you envision needing
> extensions/changes to the language to support Storm? Feel free to add a
> page to Pig's wiki with your thoughts on an approach.
> On Jul 23, 2013, at 9:52 AM, Pradeep Gollakota wrote:
> > Hi Pig Developers,
> > I wanted to reach out to you all and ask for you opinion on something.
> > As a Pig user, I have come to love Pig as a framework. Pig provides a
> > set of abstractions that make working with large datasets easy. Currently
> > Pig is only backed by hadoop. However, with the new rise of Twitter Storm
> > as a distributed real time processing engine, Pig users are missing out
> > a great opportunity to be able to work with Pig in Storm. As a user of
> > Hadoop and Storm, and keeping with the Pig philosophy of "Pigs live
> > anywhere," I'd like to get your thoughts on starting the implementation
> > a Pig backend for Storm.
> > Thanks
> > Pradeep