Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Consumer pauses when running many threads


Copy link to this message
-
Re: Consumer pauses when running many threads
Have you looked at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F?

Thanks,

Jun
On Thu, Aug 1, 2013 at 7:30 AM, Sybrandy, Casey <
[EMAIL PROTECTED]> wrote:

> Hello,
>
> We're seeing an issue running 0.7.0 where one or more of our consumers are
> pausing after about an hour when we have a lot of threads configured.  Our
> setup is as follows:
>
>
> *         4 brokers configured for 32 threads and 32 partitions on each
> broker.
>
> *         2 consumers each processing 40 streams (24 and 16).
>
> *         Zookeeper server is a CDH version that's at least 3.3.4.
>
> We were also seeing this with 3 consumers running 18 threads each.  As you
> can tell, the hardware is quite beefy and the brokers are described as
> being "bored."
>
> Outside of upgrading to 0.7.2, which we are planning on doing but can't
> yet, what else can we look into to try to resolve this or at least
> determine what's happening?
>
> Thanks.
>
> Casey
>