Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Very low volume topic

Copy link to this message
Re: Very low volume topic
My experience is solely with 0.72. More inline.

On Tue, Aug 13, 2013 at 6:47 PM, Eric Sites <[EMAIL PROTECTED]> wrote:
> Hello everyone,
> I have a very low volume topic that has 2 consumers in the same group. How do I get each consumer to only consume 1 message at a time and if the the first consumer is busy get the other consumer to consume the message?

You can't, not if you only have one partition. Each consumer is
dedicated to a single partition. Unless you deliberately tear down the
consumer and let another take over that partition (if you are using
the high-level consumer).

> Currently what I am doing is:
> First consumer connects to Kafka waits for 300 milliseconds then disconnects, waits for 10 seconds, then reconnects to see if there is a waiting message.

I don't think you need to do this. The high-level has a API that
allows you to set this timeout (I think).

> The messages kick off a long task on each server, each server can handle multiple tasks up to a limit so first I am trying to balance the tasks across multiple servers and if they are maxed out don't consume any messages.
> This will give the other server or servers a chances to pickup a message and do the task.
> I would not disconnect if I can ensure I don't have messages waiting in the queue for a server to consume them without the other servers being able to see them.

I think a better design would be to have a basic consumer that drains
the topic and hands jobs to the set of available workers. *Those*
workers perform the long-running job. Only if there are no available
workers does the consumer block. You may be trying to do too much in
the consumer.

> Thanks for the help...
> Cheers,
> Eric Sites