Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Kafka cluster with lots of topics


+
Joe Freeman 2013-11-13, 12:49
+
Neha Narkhede 2013-11-13, 14:42
+
hsy541@...> 2013-11-13, 19:50
Copy link to this message
-
Re: Kafka cluster with lots of topics
Neha Narkhede 2013-11-13, 22:55
A node being added will generally not lead to any leader election. A node
being removed will lead to leader election. You can force leader election
using the preferred replica election command -
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaLeaderElectionTool

Thanks,
Neha
On Wed, Nov 13, 2013 at 11:50 AM, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:

> I didn't see any auto leader election for adding nodes. The data are still
> skewed on the old nodes. You have to force it by running script?
>
>
> On Wed, Nov 13, 2013 at 6:41 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > At those many topics, zookeeper will be the main bottleneck. Leader
> > election process will take very long increasing the unavailability window
> > of the cluster.
> >
> > Thanks,
> > Neha
> > On Nov 13, 2013 4:49 AM, "Joe Freeman" <[EMAIL PROTECTED]> wrote:
> >
> > > Would I be correct in assuming that a Kafka cluster won't scale well to
> > > support lots (tens of millions) of topics? If I understand correctly, a
> > > node being added or removed would involve a leader election for each
> > topic,
> > > which is a relatively expensive operation?
> > >
> >
>

 
+
Edward Capriolo 2013-11-14, 04:38
+
Joe Freeman 2013-11-14, 14:19
+
Robert Rodgers 2013-11-18, 19:57