Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Limit on number of Kafka Topic


Copy link to this message
-
Re: Limit on number of Kafka Topic
We've seen big performace degradation when we tested 1024 topics, so we've
opted to go for a much smaller topic count (< 100).

On the read side, I think performance is largely driven by the ability of
the operating system to effectively cache access to #partitions*topic
files. Clearly if you divide your available memory by 1024 topics you'll
have less in the file cache per topic than you would with only 3 topics
(for example).
On Wed, Mar 27, 2013 at 11:14 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> With more topics, you may hit one of those limits: (1) # dirs allowed in a
> FS; (2) open file handlers (we keep all log segments open in the broker);
> (3) ZK nodes.
>
> Thanks,
>
> Jun
>
> On Wed, Mar 27, 2013 at 8:36 AM, Jason Huang <[EMAIL PROTECTED]>
> wrote:
>
> > Just curious - I don't see any limit in the # of topics that you can
> > have in a Kafka cluster.
> >
> > So in principle, you could have as many topics as you want, so long as
> > your hardware can keep up, right?
> >
> > Jason
> >
> > On Wed, Mar 27, 2013 at 11:33 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > At LinkedIn, our largest cluster has more than 2K topics. 5K topics
> > should
> > > be fine.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Mar 26, 2013 at 11:52 PM, Suyog Rao <[EMAIL PROTECTED]>
> wrote:
> > >
> > >> Hello,
> > >>
> > >> Wanted to check if there is any known limit on the # of topics in a
> > Kafka
> > >> cluster? I wanted to design a system which has say 5k topics and
> > >> multi-threaded consumers reading messages from these topics. Does
> anyone
> > >> have experience with such large topic size? I see in Kafka's page a
> test
> > >> for throughput w.r.t. to topic size, but the max topic size is 50.
> > >>
> > >> Thanks
> > >>
> >
>

--
Matthew Rathbone
Foursquare | Software Engineer | Server Engineering Team
[EMAIL PROTECTED] | @rathboma <http://twitter.com/rathboma> |
4sq<http://foursquare.com/rathboma>