Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # dev - Consumer lag issues


Copy link to this message
-
Consumer lag issues
Jason Rosenberg 2013-10-21, 08:50
(I've changed the subject of this thread (was "Preparing for the 0.8 final
release"))

So, I'm not sure that my issue is exactly the same as that mentioned in the
FAQ.

Anyway, in looking at the MaxLag values for several consumers (not all
consuming the same topics), it looks like there was a strange interaction.

First, a new consumer app was started, which initially had a log of MaxLag
for a period of about a day.  This lag stayed constant at about 175M, then
once this new consumer caught up, it's lag dropped down quickly toward zero.

What's interesting, is that at that time that the new consumer seemed to
have caught up, 4 other consumers which had undetectable lag, suddenly had
their lag values shoot up, taking between 2 to 4 days to recover.  What's
interesting, is that these other consumers were mostly consuming different
topics than the new consumer that triggered this lag swap.

The 1 consumer that took the longest to recover (4 days) was consuming the
same topic as the new one, and the lag value stayed roughly constant at
about 175M messages, until it recovered.

All these consumers are using their own groupid's.

Does any of this make sense?

One thing, is that I expect the new consumer which started the issues was a
super fast consumer (no downstream IO connections, etc.).

Does any of this make sense?  Could one consumer affect other consumers of
unrelated topics?  Does it make any sense that one consumer would catch up,
only to cause several other consumers to suddenly fall behind?

Thanks :)

Jason
On Sun, Oct 20, 2013 at 12:18 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> The fetch thread uses multi-fetch. Have you looked at
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whymessagesaredelayedinmyconsumer%3F
> ,
> which may be related to your issue.
>
> Thanks,
>
> Jun
>
>
> On Sun, Oct 20, 2013 at 2:48 AM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Ok,
> >
> > So here's an outline of what I think seems to have happened.
> >
> > I have a consumer, that uses a filter to consume a large number of topics
> > (e.g. several hundred).  Each topic has only a single partition.
> >
> > It normally has no trouble keeping up processing all messages on all
> > topics.   However, we had a case a couple days ago where it seemed to
> hang,
> > and not consume anything for several hours.  I restarted the consumer
> (and
> > now I've updated it from 0.8-beta1 to 0.8-latest-HEAD).  Data is flowing
> > again, but some topics are seeming to take much longer than others to
> catch
> > up.  The slow ones seem to be the topics that have more data than others
> (a
> > loose theory at present).
> >
> > Does that make sense?  If I understand things correctly, the consumer
> will
> > fetch chunks of data from each topic/partition, in order, in a big loop?
> >  So if it has caught up with most of the topics, will it waste time
> > re-polling all those (and getting nothing) before coming back to the
> topics
> > that are lagging?  Perhaps having a larger fetch size would help here?
> >
> > Jason
> >
> >
> > On Sat, Oct 19, 2013 at 6:24 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > I'll try to, next time it hangs!
> > >
> > >
> > > On Sat, Oct 19, 2013 at 4:04 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > >wrote:
> > >
> > >> Can you send around a thread dump of the halted consumer process?
> > >>
> > >>
> > >>
> > >> On Sat, Oct 19, 2013 at 12:16 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > >> wrote:
> > >>
> > >> > The latest HEAD does seem to solve one issue, where a new topic
> being
> > >> > created after the consumer is started, would not be consumed.
> > >> >
> > >> > But the bigger issue is that we have a couple different consumers
> both
> > >> > consuming the same set of topics (under different groupids), and
> > hanging
> > >> > after a while (both hanging at about the same point).  The topics in
> > >> each
> > >> > case are selected with a filter (actually a relatively large number
> of
> > >>