Home | About | Sematext search-lucene.com search-hadoop.com
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Very low volume topic


Copy link to this message
-
Re: Very low volume topic
My experience is solely with 0.72. More inline.

On Tue, Aug 13, 2013 at 6:47 PM, Eric Sites <[EMAIL PROTECTED]> wrote:

You can't, not if you only have one partition. Each consumer is
dedicated to a single partition. Unless you deliberately tear down the
consumer and let another take over that partition (if you are using
the high-level consumer).
I don't think you need to do this. The high-level has a API that
allows you to set this timeout (I think).
I think a better design would be to have a basic consumer that drains
the topic and hands jobs to the set of available workers. *Those*
workers perform the long-running job. Only if there are no available
workers does the consumer block. You may be trying to do too much in
the consumer.
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB