Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Data loss in case of request.required.acks set to -1


Copy link to this message
-
Re: Data loss in case of request.required.acks set to -1
Hi Guozhang,

I have tried with Kafka-0.8.1 after applying patch 1188 but thats not
helping in this case.

Also controlled.shutdown.enable is also not helpful in case of abnormally
shutdown (i.e. SIGKILL (-9)).

Any other suggestion?
On Thu, Dec 19, 2013 at 3:59 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:

> Yes, please go ahead.
>
>
> On Thu, Dec 19, 2013 at 2:30 AM, Hanish Bansal <
> [EMAIL PROTECTED]> wrote:
>
> > Hi Guazhang,
> >
> > Can I try it now with trunk HEAD kafka code after applying the patch
> > KAFKA-1188.patch<
> >
> https://issues.apache.org/jira/secure/attachment/12619475/KAFKA-1188.patch
> > >?
> >
> >
> >
> >
> > On Wed, Dec 18, 2013 at 9:49 PM, Guozhang Wang <[EMAIL PROTECTED]>
> wrote:
> >
> > > Kafka server's shutdown hook should capture all SIG but NOT SIGKILL
> (-9),
> > > and the controlled shut down process will not be triggered in this
> case.
> > >
> > > That said, if the other replica is in ISR then even kill -9  should not
> > > lose data. I am currently working on this JIRA that might be related if
> > > brokers are bounced iteratively:
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-1188
> > >
> > > Hanish, could you retry trunk HEAD once this one is resolved?
> > >
> > >
> > > On Wed, Dec 18, 2013 at 12:00 PM, Joe Stein <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > leader election should start for the brokers that are in the isr for
> > the
> > > > partitions that are on that replica that are leaders by the other
> > > replicas
> > > > still in the isr, and the leader failed removed from the isr.  The
> isr
> > > will
> > > > shrink for all other partitions this broker is in the isr on but not
> > the
> > > > leader.
> > > >
> > > > so lots of re-giggling and the time there is going to be related to
> how
> > > > many partitions and brokers you have.
> > > >
> > > > On Wed, Dec 18, 2013 at 2:49 PM, Robert Rodgers <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > >
> > > > > what happens if the physical machine dies or the kernel panics?
> > > > >
> > > > > On Dec 18, 2013, at 9:44 AM, Hanish Bansal <
> > > > > [EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > Yup definitely i would like to try that If
> > controlled.shutdown.enable
> > > > > > property works in case of kill -9.
> > > > > >
> > > > > > I hope that this option will be perfect.
> > > > > >
> > > > > > Thanks for quick response, really appreciate it.
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 18, 2013 at 10:52 PM, Joe Stein <
> [EMAIL PROTECTED]>
> > > > > wrote:
> > > > > >
> > > > > >> Wouldn't you want to set the controlled.shutdown.enable to true
> so
> > > the
> > > > > >> broker would do this for you before ending itself?
> > > > > >>
> > > > > >> /*******************************************
> > > > > >> Joe Stein
> > > > > >> Founder, Principal Consultant
> > > > > >> Big Data Open Source Security LLC
> > > > > >> http://www.stealth.ly
> > > > > >> Twitter: @allthingshadoop <
> http://www.twitter.com/allthingshadoop
> > >
> > > > > >> ********************************************/
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Dec 18, 2013 at 11:36 AM, pushkar priyadarshi <
> > > > > >> [EMAIL PROTECTED]> wrote:
> > > > > >>
> > > > > >>> my doubt was they are dropping off at producer level only.so
> > > > suggested
> > > > > >>> playing with paramaters like retries and backoff.ms and also
> > with
> > > > > >>> refreshinterval on producer side.
> > > > > >>>
> > > > > >>> Regards,
> > > > > >>> Pushkar
> > > > > >>>
> > > > > >>>
> > > > > >>> On Wed, Dec 18, 2013 at 10:01 PM, Guozhang Wang <
> > > [EMAIL PROTECTED]>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> Hanish,
> > > > > >>>>
> > > > > >>>> Did you "kill -9" one of the brokers only or bouncing them
> > > > > iteratively?
> > > > > >>>>
> > > > > >>>> Guozhang
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Wed, Dec 18, 2013 at 8:02 AM, Joe Stein <
> > [EMAIL PROTECTED]>
> > > > > >> wrote:
> > > > > >
*Thanks & Regards*
*Hanish Bansal*

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB