Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Consumer Lag increasing but offset not moving.


Copy link to this message
-
Re: Consumer Lag increasing but offset not moving.
On restart, the consumer starts fetching from the offset stored in ZK. So,
the issue is why the offset in ZK is not updated. Do you do auto offset
commit? If so, what's the commit interval?

Thanks,

Jun
On Fri, Aug 16, 2013 at 1:02 AM, Pavitar Singh <[EMAIL PROTECTED]> wrote:

> Hi Jun,
>
> We did some more analysis on this and we found following.
>
> Request sent/recd on Broker
>
>  FetchRequest(topic:event_queue, part:42 offset:1922674794 maxSize:3145728)
>
> While
>
>  ,1-42   1922341150       333644
>  event_queue_ip-10-31-132-218-1376638664291-c9ca8eb8-204      1922674794
>
> Consumer Offset shown in Zookeeper is 1922341150.
>
> This way we think that there is a Lag, while there is none as Consumer has
> moved ahead and asking for different Offset.
>
>
>
> On Thu, Aug 15, 2013 at 8:40 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Another possibility is that the fetch size in the consumer is smaller
> than
> > the largest message in the broker. You can use the DumpLogSegments tool
> to
> > figure out the largest message size and set fetch.size in the consumer
> > config accordingly.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, Aug 14, 2013 at 10:59 PM, Ramakrishna Mallya
> > <[EMAIL PROTECTED]>wrote:
> >
> > > Yes they are all alive and waiting to process, but fetcher is not
> pulling
> > > any messages.
> > >
> > >
> > > On Thu, Aug 15, 2013 at 9:44 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > > > Are your consumer threads all alive?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Wed, Aug 14, 2013 at 5:13 PM, Ramakrishna Mallya <
> > > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Yes that is the log for one of the consumers, but it is same for
> the
> > > > other
> > > > > consumers too that we have running for this queue. We have 4
> > consumers
> > > > > running with 300 streams each to match the partitions on the
> brokers.
> > > > >
> > > > > And yes the backlog is growing and offsets not moving. Following is
> > the
> > > > > sampling that we took at 48 intervals and the offsets are stuck.
> > > > >
> > > > >
> > > > > +----+--------+------------+
> > > > > | count  | partition name   | offset     |
> > > > > +----+--------+------------+
> > > > > | 48 | ,1-0   | 1898883492 |
> > > > > | 48 | ,1-1   | 1915507531 |
> > > > > | 48 | ,1-10  | 1918332917 |
> > > > > | 48 | ,1-100 |  236006405 |
> > > > > | 48 | ,1-101 |  236582777 |
> > > > > | 48 | ,1-102 |  235905931 |
> > > > > | 48 | ,1-103 |  236231929 |
> > > > > | 48 | ,1-104 |  237221764 |
> > > > > | 48 | ,1-105 |  238297312 |
> > > > > | 47 | ,1-106 |  235628377 |
> > > > > | 48 | ,1-107 |  234600170 |
> > > > > | 48 | ,1-108 |  238394669 |
> > > > > | 45 | ,1-109 |  239396416 |
> > > > > | 48 | ,1-11  | 1896022577 |
> > > > > | 48 | ,1-110 |  237924875 |
> > > > > | 48 | ,1-111 |  235374233 |
> > > > > | 48 | ,1-112 |  213809615 |
> > > > > | 48 | ,1-113 |  237226354 |
> > > > > | 48 | ,1-114 |  237396165 |
> > > > > | 48 | ,1-115 |  236191104 |
> > > > > | 48 | ,1-116 |  238128466 |
> > > > > | 48 | ,1-117 |  237335134 |
> > > > > | 48 | ,1-118 |  237350820 |
> > > > > | 48 | ,1-119 |  236527060 |
> > > > > | 18 | ,1-12  | 1917295864 |
> > > > > | 48 | ,1-120 |  235306049 |
> > > > > | 48 | ,1-121 |  235316391 |
> > > > > | 48 | ,1-122 |  235329292 |
> > > > > | 48 | ,1-123 |  236826301 |
> > > > > | 46 | ,1-124 |  236710764 |
> > > > > | 48 | ,1-125 |  236215243 |
> > > > > | 48 | ,1-126 |  234641764 |
> > > > > | 48 | ,1-127 |  238293647 |
> > > > > | 48 | ,1-128 |  236609422 |
> > > > > | 47 | ,1-129 |  236212259 |
> > > > > | 47 | ,1-13  | 1919322217 |
> > > > > | 48 | ,1-130 |  234732966 |
> > > > > | 46 | ,1-131 |  235335704 |
> > > > > | 48 | ,1-132 |  235608449 |
> > > > > | 48 | ,1-133 |  235595652 |
> > > > > | 48 | ,1-134 |  235550350 |
> > > > > | 48 | ,1-135 |  238191384 |
> > > > > | 48 | ,1-136 |  236527843 |
> > > > > | 48 | ,1-137 |  235194210 |
> > > > > | 48 | ,1-138 |  238141134 |
> > > > > | 48 | ,1-139 |  237072855 |