Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> High level consumer Blocked when there is still message in topic


Copy link to this message
-
Re: High level consumer Blocked when there is still message in topic
What's the max lag (reported in JMX) in the consumer? Can the consumer keep
up with the incoming data rate?

Thanks,

Jun
On Tue, Nov 12, 2013 at 7:19 PM, 李帅 <[EMAIL PROTECTED]> wrote:

> Hi,
>
>    I use Kafka 0.8 high level consumer reads message from topic
> stream, 3 replica and 10 paritions.
>
>    When I use 10 threads read the stream and runing for some time (one
> hour or one day),
>
> some threads block at m_stream.iterator().hasNext(), but the parition
> still has lots of messages.
>
>    I check consumer's fetch.message.max.bytes and broker's
> message.max.bytes, there is no
>
> message size bigger than these values.
>
>    The consumer configure is
>    props.put("zookeeper.session.timeout.ms", "4000");
>    props.put("zookeeper.sync.time.ms", "200");
>    props.put("auto.commit.interval.ms", "1000");
>
>
>    Please give me some option about how to avoid consumer block.
>
>    Is there some configure parameter can fix this problem.
>
> Thanks!
>
> XiaoTian
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB