Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Kafka cluster with lots of topics

Joe Freeman 2013-11-13, 12:49
Neha Narkhede 2013-11-13, 14:42
hsy541@...> 2013-11-13, 19:50
Neha Narkhede 2013-11-13, 22:55
Copy link to this message
Re: Kafka cluster with lots of topics
Zookeeper will not be the only problem. The first is that each topic is a
directory on the file system. Each of those is going to have files inside
it. This is going to be fairly overwhelming for the file system. Also I can
not speak for the internals but there may be cases where this many topics
allocates a big array or some other non-optimal behaviour.

Like a RDBMS with this many tables one might ask, why? Isn't there a way to
design the system multi-tennent where so many physical topics are not
On Wed, Nov 13, 2013 at 9:41 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> At those many topics, zookeeper will be the main bottleneck. Leader
> election process will take very long increasing the unavailability window
> of the cluster.
> Thanks,
> Neha
> On Nov 13, 2013 4:49 AM, "Joe Freeman" <[EMAIL PROTECTED]> wrote:
> > Would I be correct in assuming that a Kafka cluster won't scale well to
> > support lots (tens of millions) of topics? If I understand correctly, a
> > node being added or removed would involve a leader election for each
> topic,
> > which is a relatively expensive operation?
> >

Joe Freeman 2013-11-14, 14:19
Robert Rodgers 2013-11-18, 19:57