Kafka, mail # user - Re: Reading Kafka directly from Pig? - 2013-08-07, 15:20
 Search Hadoop and all its subprojects:

Switch to Plain View
+
David Arthur 2013-08-07, 14:42
+
Jun Rao 2013-08-07, 14:49
+
Russell Jurney 2013-08-07, 14:59
Copy link to this message
-
Re: Reading Kafka directly from Pig?
Right now it only terminates if SimpleConsumer hits the timeout. So, in
theory it can forever. To bound the InputFormat, I would probably add a
max time or max number of messages to consume (in addition to the timeout).

I started by looking at the Camus code, but it was easier to whip up a
simple InputFormat for testing. If this becomes a real thing, I would
probably figure out how to utilize Camus.

-David

On 8/7/13 10:49 AM, Jun Rao wrote:
 
+
David Arthur 2013-08-07, 15:28
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB