Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Mirrormaker consumer looping to offset out of range and reset offset errors


+
Rajasekar Elango 2013-10-09, 22:00
Copy link to this message
-
Re: Mirrormaker consumer looping to offset out of range and reset offset errors
Jun Rao 2013-10-10, 03:23
Not sure what the issue is. Are you using 0.8 beta1? Did you enable auto
offset commit?

Thanks,

Jun
On Wed, Oct 9, 2013 at 3:00 PM, Rajasekar Elango <[EMAIL PROTECTED]>wrote:

> We are seeing that mirrormaker consumer started looping through offset out
> of range and reset offset errors for some of partitions (2 out of 8
> partitions). The consumerOffsetChecker reported very high Lag for these 2
> partitions. Looks like this problem has started after a consumer rebalance.
> Here is log lines:
>
> 2013-10-06 06:09:59,993
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 2526006629 for partition [FunnelProto,1] out of range; reset
> offset to 2526006629
> 2013-10-06 06:09:59,993
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 2363213504 for partition [FunnelProto,3] out of range; reset
> offset to 2363213504
> 2013-10-06 06:09:59,993
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 2146256007 for partition [jmx,0] out of range; reset offset
> to 2146256007
> 2013-10-06 06:09:59,992
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 2239688 for partition [tower_timing_metrics,3] out of range;
> reset offset to 2239688
> 2013-10-06 06:09:59,889
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 1234239 for partition [agent,0] out of range; reset offset
> to 1234239
> 2013-10-06 06:09:59,889
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 2526006629 for partition [FunnelProto,1] out of range; reset
> offset to 2526006629
> 2013-10-06 06:09:59,889
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4]
> WARN  (kafka.consumer.ConsumerFetcherThread)  -
>
> [ConsumerFetcherThread-mirrormakerProd_ops-mmrs1-1-sjl.ops.sfdc.net-1380036300408-baa80a5a-0-4],
> current offset 2363213504 for partition [FunnelProto,3] out of range; reset
> offset to 2363213504
>
>
> Also, as you can it's resetting offset to same value so it's looping
> through this offset resets again and again. After we restarted our
> mirrormaker process, it started consuming from beginning topic for all
> partitions (we started received messages 7 days ) and it caught in couple
> of hours..
>
> We have couple of questions
>
> 1) What might have caused this to end up in this bad state..?
> 2) We had offset out of range problem only for 2 out of 8 partitions, but
> it started to consume from beginning for all partitions in topic after we
> restarted mirrormaker.. How problem with 2 partitions affected all other
> partitions ..?
>
>
> --
> Thanks,
> Raja.
>