Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Re: questions about ISR


Copy link to this message
-
Re: questions about ISR
Jun Rao 2013-08-27, 15:25
Look for jmx beans under kafka.server. You will see ???MaxLag and
???MinFetchRate. In the normal case, when a broker fails, the controller
will drop the failed broker out of ISR during leader election. So, the
value of replica.lag.time.max.ms doesn't matter. This value only matters
when the controller's decision is delayed somehow. In this case, having a
large replica.lag.time.max.ms may delay the committing of a message.

Thanks,

Jun
On Tue, Aug 27, 2013 at 6:37 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:

> Thanks, Jun. That is very helpful. However, I still have a couple of
> questions. "We have a min fetch rate JMX in the broker". How to
> find out how such min fetch rate is defined? And if
> replica.lag.time.max.ms is too large, what is the consequence?
>
>
>
>
> Regards,
>
> Libo
>
>
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, August 27, 2013 12:07 AM
> To: [EMAIL PROTECTED]
> Subject: Re: questions about ISR
>
> I added the following in FAQ:
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowtoreducechurnsinISR%3F
>
> Thanks,
>
> Jun
>
>
> On Mon, Aug 26, 2013 at 7:46 PM, James Wu <[EMAIL PROTECTED]> wrote:
>
> > Hi Jun,
> >
> > I am curious Yu's questions too.
> >
> > 1. What is the best practice to set replica.lag.time.max.ms &
> > replica.lag.max.messages ? As long as possible or something else ?
> >
> > 2. If the broker exceeds one of these 2 configurations, how should we
> > do to bring the broker back to ISR ? Will controller automatic cover
> > this to catch broker up, the only thing we need to do is waiting for
> > the broker back ?
> >
> > Thanks.
> >
> >
> >
> >
> > On Mon, Aug 26, 2013 at 11:15 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > That's right. You shouldn't need to restart the whole cluster for a
> > broker
> > > to rejoin ISR. Do you see many ZK session expirations in the brokers
> > > (search for "(Expired)"? If so, you may need to tune the GC on the
> > broker.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Aug 26, 2013 at 7:11 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
> > >
> > > > Hi Jun,
> > > >
> > > > Could you confirm the following?
> > > > So after a broker is out of ISR, the only way to let it go back is
> > > > to restart it.
> > > >
> > > > We should set replica.lag.time.max.ms and replica.lag.max.messages
> > > > as large as possible to avoid a broker fall outside of ISR.
> > > >
> > > > What we have experienced is that when a broker is out of ISR
> > > > frequently we need to restart the whole cluster to make it back.
> > > > That is a
> > blocking
> > > > issue
> > > > for us.
> > > >
> > > >
> > > > Regards,
> > > >
> > > > Libo
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:[EMAIL PROTECTED]]
> > > > Sent: Friday, August 23, 2013 11:41 PM
> > > > To: [EMAIL PROTECTED]
> > > > Subject: Re: questions about ISR
> > > >
> > > > When a broker is restarted, it will automatically catch up from
> > > > the
> > > leader
> > > > and will join ISR when it's caught up. Are you not seeing this
> > happening?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Fri, Aug 23, 2013 at 11:33 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > When a broker is not in a topic's ISR, will it try to catch up
> > > > > to go back to ISR itself?
> > > > > Or do we have to restart it?
> > > > >
> > > > > We can increase replica.lag.time.max.ms and
> > > > > replica.lag.max.messages to let brokers stay longer in ISR. Is
> > > > > that good practice? Still this is related to the first
> > > > > questions. We want to know what happens after a broker falls out
> of ISR and what we should do. Thanks.
> > > > >
> > > > >
> > > > > Regards,
> > > > >
> > > > > Libo
> > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Friendly regards,
> >
> > *James Wu <https://plus.google.com/u/0/100829801349304669533>
> > *
> >
>