Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Limit on number of Kafka Topic


Copy link to this message
-
Re: Limit on number of Kafka Topic
Jun Rao 2013-03-27, 16:14
With more topics, you may hit one of those limits: (1) # dirs allowed in a
FS; (2) open file handlers (we keep all log segments open in the broker);
(3) ZK nodes.

Thanks,

Jun

On Wed, Mar 27, 2013 at 8:36 AM, Jason Huang <[EMAIL PROTECTED]> wrote:

> Just curious - I don't see any limit in the # of topics that you can
> have in a Kafka cluster.
>
> So in principle, you could have as many topics as you want, so long as
> your hardware can keep up, right?
>
> Jason
>
> On Wed, Mar 27, 2013 at 11:33 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > At LinkedIn, our largest cluster has more than 2K topics. 5K topics
> should
> > be fine.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Mar 26, 2013 at 11:52 PM, Suyog Rao <[EMAIL PROTECTED]> wrote:
> >
> >> Hello,
> >>
> >> Wanted to check if there is any known limit on the # of topics in a
> Kafka
> >> cluster? I wanted to design a system which has say 5k topics and
> >> multi-threaded consumers reading messages from these topics. Does anyone
> >> have experience with such large topic size? I see in Kafka's page a test
> >> for throughput w.r.t. to topic size, but the max topic size is 50.
> >>
> >> Thanks
> >>
>