Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: possible to shutdown a consumerConnector without flushing the offset


Copy link to this message
-
RE: possible to shutdown a consumerConnector without flushing the offset
If you turn off auto.commit.enable,  that will ensure that messages are
replayed whenever a consumer starts up and rebalances.

Thanks,
Neha
On May 17, 2013 6:35 AM, "Withers, Robert" <[EMAIL PROTECTED]> wrote:

> Certainly I will try.  Our understanding is that there are 2 scenarios
> where messages could be replayed:
>
> 1. if a consumer falls over hard, there are some message consumptions
> whose offsets had not yet been flushed to zookeeper and so when a rebalance
> occurs the consumer that starts getting messages from a partition that
> flipped from the broke consumer will replay some messages.
> 2. I think a combination of a leader election and a broker failure may
> replay messages.
>
> We want to demonstrate the first, but our stats and correlation harness
> needs to keep running.  Love to demonstrate the second but egads, it's
> tricky.
>
> thanks,
> rob
> ________________________________________
> From: Neha Narkhede [[EMAIL PROTECTED]]
> Sent: Friday, May 17, 2013 7:29 AM
> To: [EMAIL PROTECTED]
> Subject: Re: possible to shutdown a consumerConnector without flushing the
> offset
>
> Can you provide more details about what you mean by measuring replay when
> you kill a consumer?
>
> Thanks,
> Neha
> On May 17, 2013 6:26 AM, "Withers, Robert" <[EMAIL PROTECTED]>
> wrote:
>
> > Would it be possible for someone to provide me with a 0.8 jar that
> > implements a ConsumerConnector.hardShutdown, which would interrupt all
> > threads yet not do a final offset flush.  We want to measure replay so we
> > want to simulate a kill -9, but we want to keep running the process to
> > flush stats and have them available locally as well.
> >
> > thanks,
> > rob
>