Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Mirrormaker consumer looping to offset out of range and reset offset errors


Copy link to this message
-
Re: Mirrormaker consumer looping to offset out of range and reset offset errors
Not sure what the issue is. Are you using 0.8 beta1? Did you enable auto
offset commit?

Thanks,

Jun
On Wed, Oct 9, 2013 at 3:00 PM, Rajasekar Elango <[EMAIL PROTECTED]>wrote:

> We are seeing that mirrormaker consumer started looping through offset out
> of range and reset offset errors for some of partitions (2 out of 8
> partitions). The consumerOffsetChecker reported very high Lag for these 2
> partitions. Looks like this problem has started after a consumer rebalance.
> Here is log lines:
>
> 2013-10-06 06:09:59,993
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 2526006629 for partition [FunnelProto,1] out of range; reset
> offset to 2526006629
> 2013-10-06 06:09:59,993
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 2363213504 for partition [FunnelProto,3] out of range; reset
> offset to 2363213504
> 2013-10-06 06:09:59,993
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 2146256007 for partition [jmx,0] out of range; reset offset
> to 2146256007
> 2013-10-06 06:09:59,992
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 2239688 for partition [tower_timing_metrics,3] out of range;
> reset offset to 2239688
> 2013-10-06 06:09:59,889
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 1234239 for partition [agent,0] out of range; reset offset
> to 1234239
> 2013-10-06 06:09:59,889
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 2526006629 for partition [FunnelProto,1] out of range; reset
> offset to 2526006629
> 2013-10-06 06:09:59,889
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 2363213504 for partition [FunnelProto,3] out of range; reset
> offset to 2363213504
>
>
> Also, as you can it's resetting offset to same value so it's looping
> through this offset resets again and again. After we restarted our
> mirrormaker process, it started consuming from beginning topic for all
> partitions (we started received messages 7 days ) and it caught in couple
> of hours..
>
> We have couple of questions
>
> 1) What might have caused this to end up in this bad state..?
> 2) We had offset out of range problem only for 2 out of 8 partitions, but
> it started to consume from beginning for all partitions in topic after we
> restarted mirrormaker.. How problem with 2 partitions affected all other
> partitions ..?
>
>
> --
> Thanks,
> Raja.
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB