Home | About | Sematext search-lucene.com search-hadoop.com search-devops.com metrics + logs = try SPM and Logsene for free
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> ETL with Kafka


+
Guy Doulberg 2013-01-06, 07:49
+
David Arthur 2013-01-06, 22:29
+
Russell Jurney 2013-01-07, 07:00
+
Guy Doulberg 2013-01-07, 07:12
+
Ken Krugler 2013-01-07, 17:57
+
Russell Jurney 2013-01-07, 20:48
+
Ken Krugler 2013-01-07, 21:51
+
Russell Jurney 2013-01-07, 22:06
Copy link to this message
-
Re: ETL with Kafka

On Jan 7, 2013, at 2:05pm, Russell Jurney wrote:
Thanks, I missed that - all I saw was the long URL to the Talend integration doc on Hortonworks.
Some Cascading integration notes, just for posterity:

Having a Kafka Tap/Scheme would make integration easy. I see there are KafkaInputFormat and KafkaOutputFormat classes in the contrib, which is great - though these would have to back-port these to the older Hadoop APIs in order to work with Cascading. Also Cascading sends all data around as the key (value is always NullWritable) whereas the Kafka input/output formats do the opposite.

Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB