Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Consumer pauses when running many threads


Copy link to this message
-
Re: Consumer pauses when running many threads
That's right. In 0.7, # partitions is per broker. However, in 0.8, #
partitions is for the whole cluster.

Thanks,

Jun
On Fri, Aug 2, 2013 at 8:13 AM, Sybrandy, Casey <
[EMAIL PROTECTED]> wrote:

> Yes, we have.  Our SA where this is occurring has been monitoring this.
>  When the consumers went down, we could see that things were lagging.
>  Yesterday, they lowered the number of threads for the consumers to six
> each and they haven't shut down yet.  There appears to still be some lag,
> but since the consumers are running, it's decreasing.
>
> A test was run with each broker configured to have 32 partitions each and
> when the number of threads across the consumers exceeds 32, then we have
> issues.  My understanding from the documentation is that when you set the
> number of partitions on a broker, it's just for that broker, correct?
>  Therefore, if we set each broker to have 32 partitions, across 4 brokers
> we should have 128 partitions per topic, correct?  In which case, we should
> be able to run 128 consumer threads with ease.
>
> Casey
>
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, August 01, 2013 11:13 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Consumer pauses when running many threads
>
> Have you looked at
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F
> ?
>
> Thanks,
>
> Jun
>
>
> On Thu, Aug 1, 2013 at 7:30 AM, Sybrandy, Casey <
> [EMAIL PROTECTED]> wrote:
>
> > Hello,
> >
> > We're seeing an issue running 0.7.0 where one or more of our consumers
> > are pausing after about an hour when we have a lot of threads
> > configured.  Our setup is as follows:
> >
> >
> > *         4 brokers configured for 32 threads and 32 partitions on each
> > broker.
> >
> > *         2 consumers each processing 40 streams (24 and 16).
> >
> > *         Zookeeper server is a CDH version that's at least 3.3.4.
> >
> > We were also seeing this with 3 consumers running 18 threads each.  As
> > you can tell, the hardware is quite beefy and the brokers are
> > described as being "bored."
> >
> > Outside of upgrading to 0.7.2, which we are planning on doing but
> > can't yet, what else can we look into to try to resolve this or at
> > least determine what's happening?
> >
> > Thanks.
> >
> > Casey
> >
>