Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Data loss in case of request.required.acks set to -1


Copy link to this message
-
Re: Data loss in case of request.required.acks set to -1
Hi Guozhang,

I have tried with Kafka-0.8.1 after applying patch 1188 but thats not
helping in this case.

Also controlled.shutdown.enable is also not helpful in case of abnormally
shutdown (i.e. SIGKILL (-9)).

Any other suggestion?
On Thu, Dec 19, 2013 at 3:59 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:

> Yes, please go ahead.
>
>
> On Thu, Dec 19, 2013 at 2:30 AM, Hanish Bansal <
> [EMAIL PROTECTED]> wrote:
>
> > Hi Guazhang,
> >
> > Can I try it now with trunk HEAD kafka code after applying the patch
> > KAFKA-1188.patch<
> >
> https://issues.apache.org/jira/secure/attachment/12619475/KAFKA-1188.patch
> > >?
> >
> >
> >
> >
> > On Wed, Dec 18, 2013 at 9:49 PM, Guozhang Wang <[EMAIL PROTECTED]>
> wrote:
> >
> > > Kafka server's shutdown hook should capture all SIG but NOT SIGKILL
> (-9),
> > > and the controlled shut down process will not be triggered in this
> case.
> > >
> > > That said, if the other replica is in ISR then even kill -9  should not
> > > lose data. I am currently working on this JIRA that might be related if
> > > brokers are bounced iteratively:
> > >
> > > https://issues.apache.org/jira/browse/KAFKA-1188
> > >
> > > Hanish, could you retry trunk HEAD once this one is resolved?
> > >
> > >
> > > On Wed, Dec 18, 2013 at 12:00 PM, Joe Stein <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > leader election should start for the brokers that are in the isr for
> > the
> > > > partitions that are on that replica that are leaders by the other
> > > replicas
> > > > still in the isr, and the leader failed removed from the isr.  The
> isr
> > > will
> > > > shrink for all other partitions this broker is in the isr on but not
> > the
> > > > leader.
> > > >
> > > > so lots of re-giggling and the time there is going to be related to
> how
> > > > many partitions and brokers you have.
> > > >
> > > > On Wed, Dec 18, 2013 at 2:49 PM, Robert Rodgers <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > >
> > > > > what happens if the physical machine dies or the kernel panics?
> > > > >
> > > > > On Dec 18, 2013, at 9:44 AM, Hanish Bansal <
> > > > > [EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > Yup definitely i would like to try that If
> > controlled.shutdown.enable
> > > > > > property works in case of kill -9.
> > > > > >
> > > > > > I hope that this option will be perfect.
> > > > > >
> > > > > > Thanks for quick response, really appreciate it.
> > > > > >
> > > > > >
> > > > > > On Wed, Dec 18, 2013 at 10:52 PM, Joe Stein <
> [EMAIL PROTECTED]>
> > > > > wrote:
> > > > > >
> > > > > >> Wouldn't you want to set the controlled.shutdown.enable to true
> so
> > > the
> > > > > >> broker would do this for you before ending itself?
> > > > > >>
> > > > > >> /*******************************************
> > > > > >> Joe Stein
> > > > > >> Founder, Principal Consultant
> > > > > >> Big Data Open Source Security LLC
> > > > > >> http://www.stealth.ly
> > > > > >> Twitter: @allthingshadoop <
> http://www.twitter.com/allthingshadoop
> > >
> > > > > >> ********************************************/
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Dec 18, 2013 at 11:36 AM, pushkar priyadarshi <
> > > > > >> [EMAIL PROTECTED]> wrote:
> > > > > >>
> > > > > >>> my doubt was they are dropping off at producer level only.so
> > > > suggested
> > > > > >>> playing with paramaters like retries and backoff.ms and also
> > with
> > > > > >>> refreshinterval on producer side.
> > > > > >>>
> > > > > >>> Regards,
> > > > > >>> Pushkar
> > > > > >>>
> > > > > >>>
> > > > > >>> On Wed, Dec 18, 2013 at 10:01 PM, Guozhang Wang <
> > > [EMAIL PROTECTED]>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> Hanish,
> > > > > >>>>
> > > > > >>>> Did you "kill -9" one of the brokers only or bouncing them
> > > > > iteratively?
> > > > > >>>>
> > > > > >>>> Guozhang
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> On Wed, Dec 18, 2013 at 8:02 AM, Joe Stein <
> > [EMAIL PROTECTED]>
> > > > > >> wrote:
> > > > > >
*Thanks & Regards*
*Hanish Bansal*