Kafka, mail # user - Re: understanding OffsetOutOfRangeException's.... - 2014-01-08, 02:47
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Plain View
Copy link to this message
-
Re: understanding OffsetOutOfRangeException's....
So, sometimes I just get the WARN from the ConsumerFetcherThread (as
previously noted, above), e.g.:

2014-01-08 02:31:47,394  WARN [ConsumerFetcherThread-myconsumerapp-11]
consumer.ConsumerFetcherThread -
[ConsumerFetcherThread-myconsumerapp-11], Current offset 16163904970
for partition [mypartition,0] out of range; reset offset to
16175326044

More recently, I see these in the following log line (not sure why I
didn't see it previously), coming from the ConsumerIterator:

2014-01-08 02:31:47,681 ERROR [myconsumerthread-0]
consumer.ConsumerIterator - consumed offset: 16163904970 doesn't match
fetch offset: 16175326044 for mytopic:0: fetched offset = 16175330598:
consumed offset = 16163904970;
 Consumer may lose data

Why would I not see this second ERROR everytime there's a
corresponding WARN on the FetcherThread for an offset reset?

Should I only be concerned about possible lost data if I see the
second ERROR log line?

Jason

On Tue, Dec 24, 2013 at 3:49 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

 
+
Jason Rosenberg 2014-01-08, 04:55
+
Jun Rao 2014-01-08, 05:08
+
Jason Rosenberg 2014-01-08, 05:30
+
Jun Rao 2014-01-08, 06:06
+
Jason Rosenberg 2014-01-08, 18:45
+
Joe Stein 2014-01-08, 19:00
+
Jason Rosenberg 2014-01-09, 00:30
+
Jun Rao 2014-01-09, 18:06
+
Jason Rosenberg 2014-01-09, 21:22
+
Jun Rao 2014-01-10, 16:06
+
Jason Rosenberg 2014-01-10, 19:34
+
Jun Rao 2014-01-11, 05:07
+
Jason Rosenberg 2014-01-13, 02:19
+
Jason Rosenberg 2013-12-23, 23:09
+
Guozhang Wang 2013-12-23, 23:46
+
Jason Rosenberg 2013-12-23, 23:53
+
Jun Rao 2013-12-24, 04:47
+
Jason Rosenberg 2013-12-24, 06:47
+
Guozhang Wang 2013-12-24, 17:51
+
Jason Rosenberg 2013-12-24, 20:50
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB