Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # dev >> Pig and Storm


Copy link to this message
-
Re: Pig and Storm
This sounds exciting.  The next question is how do you plan to do it?  Would a physical plan be translated to a Storm job (or jobs)?  Would it need a different physical plan?  Or would you just have the connection at the language layer and all the planning separate?  Do you envision needing extensions/changes to the language to support Storm?  Feel free to add a page to Pig's wiki with your thoughts on an approach.

Alan.

On Jul 23, 2013, at 9:52 AM, Pradeep Gollakota wrote:

> Hi Pig Developers,
>
> I wanted to reach out to you all and ask for you opinion on something.
>
> As a Pig user, I have come to love Pig as a framework. Pig provides a great
> set of abstractions that make working with large datasets easy. Currently
> Pig is only backed by hadoop. However, with the new rise of Twitter Storm
> as a distributed real time processing engine, Pig users are missing out on
> a great opportunity to be able to work with Pig in Storm. As a user of Pig,
> Hadoop and Storm, and keeping with the Pig philosophy of "Pigs live
> anywhere," I'd like to get your thoughts on starting the implementation of
> a Pig backend for Storm.
>
> Thanks
> Pradeep
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB