Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: possible to shutdown a consumerConnector without flushing the offset


Copy link to this message
-
RE: possible to shutdown a consumerConnector without flushing the offset
If you turn off auto.commit.enable,  that will ensure that messages are
replayed whenever a consumer starts up and rebalances.

Thanks,
Neha
On May 17, 2013 6:35 AM, "Withers, Robert" <[EMAIL PROTECTED]> wrote:

> Certainly I will try.  Our understanding is that there are 2 scenarios
> where messages could be replayed:
>
> 1. if a consumer falls over hard, there are some message consumptions
> whose offsets had not yet been flushed to zookeeper and so when a rebalance
> occurs the consumer that starts getting messages from a partition that
> flipped from the broke consumer will replay some messages.
> 2. I think a combination of a leader election and a broker failure may
> replay messages.
>
> We want to demonstrate the first, but our stats and correlation harness
> needs to keep running.  Love to demonstrate the second but egads, it's
> tricky.
>
> thanks,
> rob
> ________________________________________
> From: Neha Narkhede [[EMAIL PROTECTED]]
> Sent: Friday, May 17, 2013 7:29 AM
> To: [EMAIL PROTECTED]
> Subject: Re: possible to shutdown a consumerConnector without flushing the
> offset
>
> Can you provide more details about what you mean by measuring replay when
> you kill a consumer?
>
> Thanks,
> Neha
> On May 17, 2013 6:26 AM, "Withers, Robert" <[EMAIL PROTECTED]>
> wrote:
>
> > Would it be possible for someone to provide me with a 0.8 jar that
> > implements a ConsumerConnector.hardShutdown, which would interrupt all
> > threads yet not do a final offset flush.  We want to measure replay so we
> > want to simulate a kill -9, but we want to keep running the process to
> > flush stats and have them available locally as well.
> >
> > thanks,
> > rob
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB