Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Getting Data into Data Warehouse from Pig


+
rakesh sharma 2012-01-10, 20:42
Copy link to this message
-
Re: Getting Data into Data Warehouse from Pig
you might want to take a look at zookeeper as a coordination mechanism for
when to process what file

On Tue, Jan 10, 2012 at 12:42 PM, rakesh sharma <[EMAIL PROTECTED]
> wrote:

>
> Hi All,
> I am quite new to hadoop world and trying to work on a project using
> hadoop and pig. The data is continuously being written in hadoop by many
> producers. All producers concurrently write data to the same file for 30
> minutes duration. After 30 minutes, new file is created and they start
> writing on it. I need to run pig jobs to analyze the data from hadoop
> incrementally and push the resulted data in RDBMS. I am wondering what will
> be the right way to implement it.
> Thanks,RS
+
Dmitriy Ryaboy 2012-01-10, 23:36
+
IGZ Nick 2012-01-11, 06:03
+
IGZ Nick 2012-01-11, 06:15
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB