Kafka, mail # user - Re: ETL with Kafka - 2013-01-07, 22:21
 Search Hadoop and all its subprojects:

Switch to Plain View
+
Guy Doulberg 2013-01-06, 07:49
+
David Arthur 2013-01-06, 22:29
+
Russell Jurney 2013-01-07, 07:00
+
Guy Doulberg 2013-01-07, 07:12
+
Ken Krugler 2013-01-07, 17:57
+
Russell Jurney 2013-01-07, 20:48
+
Ken Krugler 2013-01-07, 21:51
+
Russell Jurney 2013-01-07, 22:06
Copy link to this message
-
Re: ETL with Kafka

On Jan 7, 2013, at 2:05pm, Russell Jurney wrote:
Thanks, I missed that - all I saw was the long URL to the Talend integration doc on Hortonworks.
Some Cascading integration notes, just for posterity:

Having a Kafka Tap/Scheme would make integration easy. I see there are KafkaInputFormat and KafkaOutputFormat classes in the contrib, which is great - though these would have to back-port these to the older Hadoop APIs in order to work with Cascading. Also Cascading sends all data around as the key (value is always NullWritable) whereas the Kafka input/output formats do the opposite.

Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB