Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Number of consumers limited to number of partitions


+
Vaibhav Puranik 2012-10-11, 15:20
+
Neha Narkhede 2012-10-11, 15:27
+
Jay Kreps 2012-10-11, 15:41
Copy link to this message
-
Re: Number of consumers limited to number of partitions
Thanks Neha and Jay.
It makes sense.

Regards,
Vaibhav

On Thu, Oct 11, 2012 at 8:41 AM, Jay Kreps <[EMAIL PROTECTED]> wrote:

> Yes, the reason for this is that if two consumers consumed the same
> partition internally we would have to keep track of per-message consumption
> (which is expensive) and the ordering guarantee would be broken (since the
> actual processing in the two consumer processes might happen out of order).
> Since you can always have more partitions this should not be a limitation,
> just a parameter to be aware of.
>
> -Jay
>
> On Thu, Oct 11, 2012 at 8:20 AM, Vaibhav Puranik <[EMAIL PROTECTED]>
> wrote:
>
> > Hi all,
> >
> > We have a topic with three partitions - one on each broker (Kafka 0.7.1).
> > Using ConsumerConnector I noticed that I can consume messages from three
> > consumers at most. When I start fourth one, one of the older one stops.
> >
> > Is this as designed? If I want more parallelism, do I need to increase
> > number of partitions?
> >
> > Regards,
> > Vaibhav
> > GumGum
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB