Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka cluster with lots of topics


+
Joe Freeman 2013-11-13, 12:49
+
Neha Narkhede 2013-11-13, 14:42
+
hsy541@...> 2013-11-13, 19:50
Copy link to this message
-
Re: Kafka cluster with lots of topics
A node being added will generally not lead to any leader election. A node
being removed will lead to leader election. You can force leader election
using the preferred replica election command -
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaLeaderElectionTool

Thanks,
Neha
On Wed, Nov 13, 2013 at 11:50 AM, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:

> I didn't see any auto leader election for adding nodes. The data are still
> skewed on the old nodes. You have to force it by running script?
>
>
> On Wed, Nov 13, 2013 at 6:41 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > At those many topics, zookeeper will be the main bottleneck. Leader
> > election process will take very long increasing the unavailability window
> > of the cluster.
> >
> > Thanks,
> > Neha
> > On Nov 13, 2013 4:49 AM, "Joe Freeman" <[EMAIL PROTECTED]> wrote:
> >
> > > Would I be correct in assuming that a Kafka cluster won't scale well to
> > > support lots (tens of millions) of topics? If I understand correctly, a
> > > node being added or removed would involve a leader election for each
> > topic,
> > > which is a relatively expensive operation?
> > >
> >
>

 
+
Edward Capriolo 2013-11-14, 04:38
+
Joe Freeman 2013-11-14, 14:19
+
Robert Rodgers 2013-11-18, 19:57
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB