Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Consumer Lag increasing but offset not moving.


Copy link to this message
-
Re: Consumer Lag increasing but offset not moving.
Another possibility is that the fetch size in the consumer is smaller than
the largest message in the broker. You can use the DumpLogSegments tool to
figure out the largest message size and set fetch.size in the consumer
config accordingly.

Thanks,

Jun
On Wed, Aug 14, 2013 at 10:59 PM, Ramakrishna Mallya
<[EMAIL PROTECTED]>wrote:

> Yes they are all alive and waiting to process, but fetcher is not pulling
> any messages.
>
>
> On Thu, Aug 15, 2013 at 9:44 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Are your consumer threads all alive?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, Aug 14, 2013 at 5:13 PM, Ramakrishna Mallya <
> [EMAIL PROTECTED]
> > >wrote:
> >
> > > Yes that is the log for one of the consumers, but it is same for the
> > other
> > > consumers too that we have running for this queue. We have 4 consumers
> > > running with 300 streams each to match the partitions on the brokers.
> > >
> > > And yes the backlog is growing and offsets not moving. Following is the
> > > sampling that we took at 48 intervals and the offsets are stuck.
> > >
> > >
> > > +----+--------+------------+
> > > | count  | partition name   | offset     |
> > > +----+--------+------------+
> > > | 48 | ,1-0   | 1898883492 |
> > > | 48 | ,1-1   | 1915507531 |
> > > | 48 | ,1-10  | 1918332917 |
> > > | 48 | ,1-100 |  236006405 |
> > > | 48 | ,1-101 |  236582777 |
> > > | 48 | ,1-102 |  235905931 |
> > > | 48 | ,1-103 |  236231929 |
> > > | 48 | ,1-104 |  237221764 |
> > > | 48 | ,1-105 |  238297312 |
> > > | 47 | ,1-106 |  235628377 |
> > > | 48 | ,1-107 |  234600170 |
> > > | 48 | ,1-108 |  238394669 |
> > > | 45 | ,1-109 |  239396416 |
> > > | 48 | ,1-11  | 1896022577 |
> > > | 48 | ,1-110 |  237924875 |
> > > | 48 | ,1-111 |  235374233 |
> > > | 48 | ,1-112 |  213809615 |
> > > | 48 | ,1-113 |  237226354 |
> > > | 48 | ,1-114 |  237396165 |
> > > | 48 | ,1-115 |  236191104 |
> > > | 48 | ,1-116 |  238128466 |
> > > | 48 | ,1-117 |  237335134 |
> > > | 48 | ,1-118 |  237350820 |
> > > | 48 | ,1-119 |  236527060 |
> > > | 18 | ,1-12  | 1917295864 |
> > > | 48 | ,1-120 |  235306049 |
> > > | 48 | ,1-121 |  235316391 |
> > > | 48 | ,1-122 |  235329292 |
> > > | 48 | ,1-123 |  236826301 |
> > > | 46 | ,1-124 |  236710764 |
> > > | 48 | ,1-125 |  236215243 |
> > > | 48 | ,1-126 |  234641764 |
> > > | 48 | ,1-127 |  238293647 |
> > > | 48 | ,1-128 |  236609422 |
> > > | 47 | ,1-129 |  236212259 |
> > > | 47 | ,1-13  | 1919322217 |
> > > | 48 | ,1-130 |  234732966 |
> > > | 46 | ,1-131 |  235335704 |
> > > | 48 | ,1-132 |  235608449 |
> > > | 48 | ,1-133 |  235595652 |
> > > | 48 | ,1-134 |  235550350 |
> > > | 48 | ,1-135 |  238191384 |
> > > | 48 | ,1-136 |  236527843 |
> > > | 48 | ,1-137 |  235194210 |
> > > | 48 | ,1-138 |  238141134 |
> > > | 48 | ,1-139 |  237072855 |
> > > | 47 | ,1-14  | 1921374677 |
> > > | 48 | ,1-140 |  219005350 |
> > > | 48 | ,1-141 |  236705945 |
> > > | 48 | ,1-142 |  235678986 |
> > > | 48 | ,1-143 |  217891350 |
> > > | 47 | ,1-144 |  238338081 |
> > > | 48 | ,1-145 |  237575489 |
> > > | 48 | ,1-146 |  236203967 |
> > > | 48 | ,1-147 |  237968410 |
> > > | 48 | ,1-148 |  233984418 |
> > > | 48 | ,1-149 |  238502295 |
> > > | 48 | ,1-15  | 1903106145 |
> > > | 48 | ,1-150 |   13499853 |
> > > | 48 | ,1-151 |   13666233 |
> > > | 48 | ,1-152 |   13670468 |
> > > | 48 | ,1-153 |   13340356 |
> > > | 44 | ,1-154 |   13406522 |
> > > | 48 | ,1-155 |   13226218 |
> > > | 48 | ,1-156 |   13779615 |
> > > | 48 | ,1-157 |   13516640 |
> > > | 48 | ,1-158 |   13523236 |
> > > | 48 | ,1-159 |   13705191 |
> > > | 48 | ,1-16  | 1919070142 |
> > > | 47 | ,1-160 |   13397958 |
> > > | 48 | ,1-161 |   13504273 |
> > > | 48 | ,1-162 |   13475059 |
> > > | 47 | ,1-163 |   13452042 |
> > > | 46 | ,1-164 |   13857996 |
> > > | 48 | ,1-165 |   13613007 |
> > > | 48 | ,1-166 |   13577619 |
> > > | 48 | ,1-167 |   13369409 |
> > > | 47 | ,1-168 |   13459302 |
> > > | 48 | ,1-169 |   14036491 |
> > > | 48 | ,1-17  | 1921745019 |

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB