Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: Consumer pauses when running many threads


Copy link to this message
-
Re: Consumer pauses when running many threads
Have you looked at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F?

Thanks,

Jun
On Thu, Aug 1, 2013 at 7:30 AM, Sybrandy, Casey <
[EMAIL PROTECTED]> wrote:

> Hello,
>
> We're seeing an issue running 0.7.0 where one or more of our consumers are
> pausing after about an hour when we have a lot of threads configured.  Our
> setup is as follows:
>
>
> *         4 brokers configured for 32 threads and 32 partitions on each
> broker.
>
> *         2 consumers each processing 40 streams (24 and 16).
>
> *         Zookeeper server is a CDH version that's at least 3.3.4.
>
> We were also seeing this with 3 consumers running 18 threads each.  As you
> can tell, the hardware is quite beefy and the brokers are described as
> being "bored."
>
> Outside of upgrading to 0.7.2, which we are planning on doing but can't
> yet, what else can we look into to try to resolve this or at least
> determine what's happening?
>
> Thanks.
>
> Casey
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB