Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> ETL with Kafka


Copy link to this message
-
Re: ETL with Kafka
Hi Guy,

On Jan 6, 2013, at 11:11pm, Guy Doulberg wrote:
Interesting - we build ETLs on top of Hadoop using Cascading (open source workflow API), which has a lot of what it calls "Taps" for connecting to data sources and sinks.

But I haven't heard of a Kafka Tap. Should be possible to implement, though.

One issue is that Hadoop is batch oriented, so there's a bit of an impedance mismatch when you've got a streaming data source, but from experience it's possible to get that to work.

Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB