Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Kafka consumer not consuming events


Copy link to this message
-
Re: Kafka consumer not consuming events
For those partitions that are lagging, do you see fetch requests in the log?

Thanks,

Jun
On Fri, Jul 19, 2013 at 12:30 AM, Nihit Purwar <[EMAIL PROTECTED]> wrote:

> Hello Jun,
>
> Sorry for the delay in getting the logs.
> Here are the 3 logs from the 3 servers with trace level as suggested:
>
>
> https://docs.google.com/file/d/0B5etsywBa-bkQnBESUJzNV9yRWc/edit?usp=sharing
>
> Please have a look and let us know if you need anything else to further
> debug this problem.
>
> Thanks,
> Nihit
>
> On 11-Jul-2013, at 4:41 PM, Nihit Purwar <[EMAIL PROTECTED]> wrote:
>
> > Hi Jun,
> >
> > I did put in only one topic while starting the consumer and have used
> the same API "createMessageStreams".
> > As for the trace level logs of kafka consumer, we will send that to you
> soon.
> >
> > Thanks again for replying.
> >
> > Nihit
> >
> > On 10-Jul-2013, at 10:38 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> >> Also, just so that we are on the same page. I assume that you used the
> >> following api. Did you just put in one topic in the topicCountMap?
> >> def createMessageStreams(topicCountMap: Map[String,Int]): Map[String,
> >> List[KafkaStream[Array[Byte],Array[Byte]]]]
> >>
> >> Thank,
> >>
> >> Jun
> >>
> >>
> >> On Wed, Jul 10, 2013 at 8:30 AM, Nihit Purwar <[EMAIL PROTECTED]>
> wrote:
> >>
> >>> Hi Jun,
> >>>
> >>> Thanks for helping out so far.
> >>>
> >>> As per your explanation we are doing exactly as you have mentioned in
> your
> >>> workaround below.
> >>>> A workaround is to use different consumer connectors, each consuming a
> >>>> single topic.
> >>>
> >>>
> >>> Here is the problem...
> >>>
> >>> We have a topic which gets a lot of events (around a million in a
> day), so
> >>> this topic on the server has a high number of partitions, and we have
> >>> dedicated consumers only listening to this topic and the processing
> time is
> >>> in the order of 15-30 millis. So we are assured that our consumers are
> not
> >>> slow in processing.
> >>>
> >>> Every now then, it so happens, that our consumers threads stalls and do
> >>> not receive any events (as suggested in my previous email with the
> thread
> >>> stack on idle threads) even though we can see the offset lag
> increasing for
> >>> the consumers.
> >>>
> >>> We also noticed that if we force rebalance the consumers (either by
> >>> starting a new consumer or killing an existing one) data starts to
> flow in
> >>> again to these consumer threads. The consumers remains stable
> (processing
> >>> events) for about 20-30 mins before the threads go idle again and the
> >>> backlog starts growing. This happens in a cycle for us and we are not
> able
> >>> to figure out the cause for events not flowing in.
> >>>
> >>> As a side note, we are also monitoring the GC cycles and there are
> hardly
> >>> any.
> >>>
> >>> Please let us know if you need any additional details.
> >>>
> >>> Thanks
> >>> Nihit.
> >>>
> >>>
> >>> On 10-Jul-2013, at 8:30 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >>>
> >>>> Ok. One of the issues is that when you have a consumer that consumes
> >>>> multiple topics, if one of the consumer threads is slow in consuming
> >>>> messages from one topic, it can block the consumption of other
> consumer
> >>>> threads. This is because we use a shared fetcher to fetch all topics.
> >>> There
> >>>> is an in-memory queue per topic. If one of the queues is full, the
> >>> fetcher
> >>>> will block and can't put the data into other queues.
> >>>>
> >>>> A workaround is to use different consumer connectors, each consuming a
> >>>> single topic.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Jun
> >>>>
> >>>>
> >>>> On Tue, Jul 9, 2013 at 11:12 PM, Nihit Purwar <[EMAIL PROTECTED]>
> >>> wrote:
> >>>>
> >>>>> Hi Jun,
> >>>>>
> >>>>> Please see my comments inline again :)
> >>>>>
> >>>>> On 10-Jul-2013, at 9:13 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >>>>>
> >>>>>> This indicates our in-memory queue is empty. So the consumer thread
> is
> >>>>>> blocked.
> >>>>>
> >>>>> What should we do about this.