Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Is there a limitation on the number of simultaneous consumer connections to the same topic


+
Ming Li 2013-05-13, 10:16
+
Andrea Gazzarini 2013-05-13, 10:44
+
Ming Li 2013-05-13, 12:49
Copy link to this message
-
Re: Is there a limitation on the number of simultaneous consumer connections to the same topic
Chris Curtin 2013-05-13, 13:10
Yes. However be aware that starting and stopping processes will cause a
rebalance of the consumers, so your code may find itself receiving events
from a different partition suddenly (so don't assume the partition you are
reading isn't going to change!)  Also as things are starting up you may
find a process receives many partitions at first, but as the other
processes are started the partitions get reassigned.

Finally, running more processes than partitions will mean those other
processes are idle.
On Mon, May 13, 2013 at 8:49 AM, Ming Li <[EMAIL PROTECTED]> wrote:

> Hi Andrea,
>
> Thanks for your reply~~ you mean, it is no difference between
> having N threads share the same ConsumerConnector created by
> Consumer.createJavaConsumerConnector,
> and
> having N consumer process which has its own ConsumerConnector in every one
> of them?
>
> Best Regards,
> Li Ming
>
>
>
> On Mon, May 13, 2013 at 6:43 PM, Andrea Gazzarini <
> [EMAIL PROTECTED]> wrote:
>
> > It shouldn't.
> > Creating several listener / consumer processes belonging to the same
> group
> > means you are working with a point-to-point message channel so incoming
> > messages will be delivered only to one consumer.
> >
> > Maybe I'm wrong but I believe in that scenario there's no difference
> (from
> > broker perspective) between threads and processes.
> >
> > Regards,
> > Andrea
> >
> >
> > On 05/13/2013 12:15 PM, Ming Li wrote:
> >
> >> Hi,
> >>
> >> Does Kafka have a limitation on the simultaneous connections (created
> with
> >> Consumer.**createJavaConsumerConnector) for the same topic within the
> >> same
> >> group?
> >>
> >> My scenario is I need to consume a topic from different process (not
> >> thread), so I need to create lots of high level consumers.
> >>
> >>
> >> Best Regards,
> >> Li Ming
> >>
> >>
> >
>

 
+
Ming Li 2013-05-14, 03:09
+
Jun Rao 2013-05-13, 15:12