Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka crashed after multiple topics were added


Copy link to this message
-
Re: Kafka crashed after multiple topics were added
Technically it is
  topics * partitions * replicas * 2 (index file and log file) + #open
sockets

-Jay
On Thu, Aug 15, 2013 at 11:49 AM, Vadim Keylis <[EMAIL PROTECTED]>wrote:

> Good Morning Joel. Just to understand clearly how to predict number of open
> files kept by kafka.
>
> That is calculated by  multiplying number of topics * number of partitions
> * number of replicas. In our case it would be 150 * 36 * 3. Am I correct?
> How number of producers and consumers will influence/impact that
> calculation? Is it advisable to have less partition? Does 36 partition
> sounds reasonable?
>
> Thanks so much in advance
>
>
>
>
> On Wed, Aug 14, 2013 at 9:27 AM, Joel Koshy <[EMAIL PROTECTED]> wrote:
>
> > We use 30k as the limit. It is largely driven by the number of partitions
> > (including replicas), retention period and number of
> > simultaneous producers/consumers.
> >
> > In your case it seems you have 150 topics, 36 partitions, 3x replication
> -
> > with that configuration you will definitely need to up your file handle
> > limit.
> >
> > Thanks,
> >
> > Joel
> >
> > On Wednesday, August 14, 2013, Vadim Keylis wrote:
> >
> > > Good morning Jun. Correction in terms of open file handler limit. I was
> > > wrong. I re-ran the command  ulimit -Hn and it shows 10240. Which
> brings
> > to
> > > the next question. How appropriately calculate open files handler
> > required
> > > by Kafka? What is your guys settings for this field?
> > >
> > > Thanks,
> > > Vadim
> > >
> > >
> > >
> > > On Wed, Aug 14, 2013 at 8:19 AM, Vadim Keylis <[EMAIL PROTECTED]
> > <javascript:;>>
> > > wrote:
> > >
> > > > Good morning Jun. We are using Kafka 0.8 that I built from trunk in
> > June
> > > > or early July. I forgot to mention that running ulimit on the hosts
> > shows
> > > > open file handler set to unlimited. What are the ways to recover from
> > > last
> > > > error and restart Kafka ? How can I delete topic with Kafka service
> on
> > > all
> > > > host down? How many topics can Kafka support to prevent to many open
> > file
> > > > exception? What did you set open file handler limit in your cluster?
> > > >
> > > > Thanks so much,
> > > > Vadim
> > > >
> > > > Sent from my iPhone
> > > >
> > > > On Aug 14, 2013, at 7:38 AM, Jun Rao <[EMAIL PROTECTED]<javascript:;>>
> > > wrote:
> > > >
> > > > > The first error is caused by too many open file handlers. Kafka
> keeps
> > > > each
> > > > > of the segment files open on the broker. So, the more
> > topics/partitions
> > > > you
> > > > > have, the more file handlers you need. You probably need to
> increase
> > > the
> > > > > open file handler limit and also monitor the # of open file
> handlers
> > so
> > > > > that you can get an alert when it gets close to the limit.
> > > > >
> > > > > Not sure why you get the second error on restart. Are you using the
> > 0.8
> > > > > beta1 release?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Aug 13, 2013 at 11:04 PM, Vadim Keylis <
> > [EMAIL PROTECTED]<javascript:;>
> > > > >wrote:
> > > > >
> > > > >> We have 3 node kafka cluster. I initially created 4 topics.
> > > > >> I wrote small shell script to create 150 topics.
> > > > >>
> > > > >> TOPICS=$(< $1)
> > > > >> for topic in $TOPICS
> > > > >> do
> > > > >>   echo "/usr/local/kafka/bin/kafka-create-topic.sh --replica 3
> > --topic
> > > > >> $topic --zookeeper $2:2181/kafka --partition 36"
> > > > >>   /usr/local/kafka/bin/kafka-create-topic.sh --replica 3 --topic
> > > $topic
> > > > >> --zookeeper $2:2181/kafka --partition 36
> > > > >> done
> > > > >>
> > > > >> 10 minutes later I see messages like this
> > > > >> [2013-08-13 11:43:58,944] INFO [ReplicaFetcherManager on broker 7]
> > > > Removing
> > > > >> fetcher for partition [m3_registration,0]
> > > > >> (kafka.server.ReplicaFetcherManager) followed by
> > > > >> [2013-08-13 11:44:00,067] WARN [ReplicaFetcherThread-0-8], error
> for
> > > > >> partition [m3_registration,22] to broker 8

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB