-RE: possible to shutdown a consumerConnector without flushing the offset
Neha Narkhede 2013-05-17, 13:40
If you turn off auto.commit.enable, that will ensure that messages are
replayed whenever a consumer starts up and rebalances.
On May 17, 2013 6:35 AM, "Withers, Robert" <[EMAIL PROTECTED]> wrote:
> Certainly I will try. Our understanding is that there are 2 scenarios
> where messages could be replayed:
> 1. if a consumer falls over hard, there are some message consumptions
> whose offsets had not yet been flushed to zookeeper and so when a rebalance
> occurs the consumer that starts getting messages from a partition that
> flipped from the broke consumer will replay some messages.
> 2. I think a combination of a leader election and a broker failure may
> replay messages.
> We want to demonstrate the first, but our stats and correlation harness
> needs to keep running. Love to demonstrate the second but egads, it's
> From: Neha Narkhede [[EMAIL PROTECTED]]
> Sent: Friday, May 17, 2013 7:29 AM
> To: [EMAIL PROTECTED]
> Subject: Re: possible to shutdown a consumerConnector without flushing the
> Can you provide more details about what you mean by measuring replay when
> you kill a consumer?
> On May 17, 2013 6:26 AM, "Withers, Robert" <[EMAIL PROTECTED]>
> > Would it be possible for someone to provide me with a 0.8 jar that
> > implements a ConsumerConnector.hardShutdown, which would interrupt all
> > threads yet not do a final offset flush. We want to measure replay so we
> > want to simulate a kill -9, but we want to keep running the process to
> > flush stats and have them available locally as well.
> > thanks,
> > rob