Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Re: Consumer pauses when running many threads


+
Jun Rao 2013-08-01, 15:13
Copy link to this message
-
Re: Consumer pauses when running many threads
That's right. In 0.7, # partitions is per broker. However, in 0.8, #
partitions is for the whole cluster.

Thanks,

Jun
On Fri, Aug 2, 2013 at 8:13 AM, Sybrandy, Casey <
[EMAIL PROTECTED]> wrote:

> Yes, we have.  Our SA where this is occurring has been monitoring this.
>  When the consumers went down, we could see that things were lagging.
>  Yesterday, they lowered the number of threads for the consumers to six
> each and they haven't shut down yet.  There appears to still be some lag,
> but since the consumers are running, it's decreasing.
>
> A test was run with each broker configured to have 32 partitions each and
> when the number of threads across the consumers exceeds 32, then we have
> issues.  My understanding from the documentation is that when you set the
> number of partitions on a broker, it's just for that broker, correct?
>  Therefore, if we set each broker to have 32 partitions, across 4 brokers
> we should have 128 partitions per topic, correct?  In which case, we should
> be able to run 128 consumer threads with ease.
>
> Casey
>
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, August 01, 2013 11:13 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Consumer pauses when running many threads
>
> Have you looked at
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F
> ?
>
> Thanks,
>
> Jun
>
>
> On Thu, Aug 1, 2013 at 7:30 AM, Sybrandy, Casey <
> [EMAIL PROTECTED]> wrote:
>
> > Hello,
> >
> > We're seeing an issue running 0.7.0 where one or more of our consumers
> > are pausing after about an hour when we have a lot of threads
> > configured.  Our setup is as follows:
> >
> >
> > *         4 brokers configured for 32 threads and 32 partitions on each
> > broker.
> >
> > *         2 consumers each processing 40 streams (24 and 16).
> >
> > *         Zookeeper server is a CDH version that's at least 3.3.4.
> >
> > We were also seeing this with 3 consumers running 18 threads each.  As
> > you can tell, the hardware is quite beefy and the brokers are
> > described as being "bored."
> >
> > Outside of upgrading to 0.7.2, which we are planning on doing but
> > can't yet, what else can we look into to try to resolve this or at
> > least determine what's happening?
> >
> > Thanks.
> >
> > Casey
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB