Kafka, mail # user - Re: ETL with Kafka - 2013-01-07, 22:21
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Plain View
Guy Doulberg 2013-01-06, 07:49
David Arthur 2013-01-06, 22:29
Russell Jurney 2013-01-07, 07:00
Guy Doulberg 2013-01-07, 07:12
Ken Krugler 2013-01-07, 17:57
Russell Jurney 2013-01-07, 20:48
Ken Krugler 2013-01-07, 21:51
Russell Jurney 2013-01-07, 22:06
Copy link to this message
Re: ETL with Kafka

On Jan 7, 2013, at 2:05pm, Russell Jurney wrote:
Thanks, I missed that - all I saw was the long URL to the Talend integration doc on Hortonworks.
Some Cascading integration notes, just for posterity:

Having a Kafka Tap/Scheme would make integration easy. I see there are KafkaInputFormat and KafkaOutputFormat classes in the contrib, which is great - though these would have to back-port these to the older Hadoop APIs in order to work with Cascading. Also Cascading sends all data around as the key (value is always NullWritable) whereas the Kafka input/output formats do the opposite.

Ken Krugler
+1 530-210-6378
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB