Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Limit on number of Kafka Topic


+
Suyog Rao 2013-03-27, 06:52
+
Jun Rao 2013-03-27, 15:33
+
Jason Huang 2013-03-27, 15:37
Copy link to this message
-
Re: Limit on number of Kafka Topic
With more topics, you may hit one of those limits: (1) # dirs allowed in a
FS; (2) open file handlers (we keep all log segments open in the broker);
(3) ZK nodes.

Thanks,

Jun

On Wed, Mar 27, 2013 at 8:36 AM, Jason Huang <[EMAIL PROTECTED]> wrote:

> Just curious - I don't see any limit in the # of topics that you can
> have in a Kafka cluster.
>
> So in principle, you could have as many topics as you want, so long as
> your hardware can keep up, right?
>
> Jason
>
> On Wed, Mar 27, 2013 at 11:33 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > At LinkedIn, our largest cluster has more than 2K topics. 5K topics
> should
> > be fine.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Mar 26, 2013 at 11:52 PM, Suyog Rao <[EMAIL PROTECTED]> wrote:
> >
> >> Hello,
> >>
> >> Wanted to check if there is any known limit on the # of topics in a
> Kafka
> >> cluster? I wanted to design a system which has say 5k topics and
> >> multi-threaded consumers reading messages from these topics. Does anyone
> >> have experience with such large topic size? I see in Kafka's page a test
> >> for throughput w.r.t. to topic size, but the max topic size is 50.
> >>
> >> Thanks
> >>
>

 
+
Matthew Rathbone 2013-03-27, 16:49
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB