Search Hadoop and all its sub project:
Sort:

Switch to Threaded View
Subject: Re: Reading Kafka directly from Pig?
Right now it only terminates if SimpleConsumer hits the timeout. So, in
theory it can forever. To bound the InputFormat, I would probably add a
max time or max number of messages to consume (in addition to the timeout).

I started by looking at the Camus code, but it was easier to whip up a
simple InputFormat for testing. If this becomes a real thing, I would
probably figure out how to utilize Camus.

-David

On 8/7/13 10:49 AM, Jun Rao wrote:
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB