Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Pig and Storm

Copy link to this message
Re: Pig and Storm
This sounds exciting.  The next question is how do you plan to do it?  Would a physical plan be translated to a Storm job (or jobs)?  Would it need a different physical plan?  Or would you just have the connection at the language layer and all the planning separate?  Do you envision needing extensions/changes to the language to support Storm?  Feel free to add a page to Pig's wiki with your thoughts on an approach.


On Jul 23, 2013, at 9:52 AM, Pradeep Gollakota wrote:

> Hi Pig Developers,
> I wanted to reach out to you all and ask for you opinion on something.
> As a Pig user, I have come to love Pig as a framework. Pig provides a great
> set of abstractions that make working with large datasets easy. Currently
> Pig is only backed by hadoop. However, with the new rise of Twitter Storm
> as a distributed real time processing engine, Pig users are missing out on
> a great opportunity to be able to work with Pig in Storm. As a user of Pig,
> Hadoop and Storm, and keeping with the Pig philosophy of "Pigs live
> anywhere," I'd like to get your thoughts on starting the implementation of
> a Pig backend for Storm.
> Thanks
> Pradeep