Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Number of consumers limited to number of partitions


Copy link to this message
-
Re: Number of consumers limited to number of partitions
Thanks Neha and Jay.
It makes sense.

Regards,
Vaibhav

On Thu, Oct 11, 2012 at 8:41 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Yes, the reason for this is that if two consumers consumed the same
> partition internally we would have to keep track of per-message consumption
> (which is expensive) and the ordering guarantee would be broken (since the
> actual processing in the two consumer processes might happen out of order).
> Since you can always have more partitions this should not be a limitation,
> just a parameter to be aware of.
>
> -Jay
>
> On Thu, Oct 11, 2012 at 8:20 AM, Vaibhav Puranik <[EMAIL PROTECTED]>
> wrote:
>
> > Hi all,
> >
> > We have a topic with three partitions - one on each broker (Kafka 0.7.1).
> > Using ConsumerConnector I noticed that I can consume messages from three
> > consumers at most. When I start fourth one, one of the older one stops.
> >
> > Is this as designed? If I want more parallelism, do I need to increase
> > number of partitions?
> >
> > Regards,
> > Vaibhav
> > GumGum
> >
>