Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - High level consumer Blocked when there is still message in topic


Copy link to this message
-
Re: High level consumer Blocked when there is still message in topic
Jun Rao 2013-11-13, 05:28
What's the max lag (reported in JMX) in the consumer? Can the consumer keep
up with the incoming data rate?

Thanks,

Jun
On Tue, Nov 12, 2013 at 7:19 PM, 李帅 <[EMAIL PROTECTED]> wrote:

> Hi,
>
>    I use Kafka 0.8 high level consumer reads message from topic
> stream, 3 replica and 10 paritions.
>
>    When I use 10 threads read the stream and runing for some time (one
> hour or one day),
>
> some threads block at m_stream.iterator().hasNext(), but the parition
> still has lots of messages.
>
>    I check consumer's fetch.message.max.bytes and broker's
> message.max.bytes, there is no
>
> message size bigger than these values.
>
>    The consumer configure is
>    props.put("zookeeper.session.timeout.ms", "4000");
>    props.put("zookeeper.sync.time.ms", "200");
>    props.put("auto.commit.interval.ms", "1000");
>
>
>    Please give me some option about how to avoid consumer block.
>
>    Is there some configure parameter can fix this problem.
>
> Thanks!
>
> XiaoTian
>