Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Why do we need reconnect with an interval?


Copy link to this message
-
Re: Why do we need reconnect with an interval?
Jun Rao 2012-11-19, 15:45
If use you zk.connect in the producer, you can leave reconnect.interval as
maxInt since we do load balancing using ZK. If you use a single VIP in
broker.list in the producer,  reconnect.interval will help load balancing.

Thanks,

Jun

On Sun, Nov 18, 2012 at 10:15 PM, Bae, Jae Hyeon <[EMAIL PROTECTED]> wrote:

> Thank you for your answer.
>
> SyncProducer doesn't change its host and port when reconnecting. So, I
> don't understand why reconnection can be helpful for load balancing.
> When the new broker is registered in the zookeeper, it will be added
> in the ProducerPool by producerCbk, which does not seem related with
> balancing.
>
> So, is it totally safe to change reconnect.interval as
> Integer.MAX_VALUE and reconnect.time.interval.ms as -1 in the
> environment where the broker can be newly created?
>
> On Sun, Nov 18, 2012 at 9:57 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > In addition to that, reconnect interval is useful for load balancing. If
> a
> > broker is restarted, initially, no existing producers are connected to
> it.
> > Reconnect interval gives the producer a chance to connect to the new
> broker
> > periodically.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Nov 18, 2012 at 3:47 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
> >
> >> Hi,
> >>
> >> This config is really relevant to 0.7 Kafka users using a VIP to talk
> >> to the Kafka cluster. VIP has a timeout for every TCP connection,
> >> where it cuts the connection that has been idle for a certain time.
> >> Most VIPs are configured to not send TCP RST back to the client, so
> >> the client does not know that the tcp connection is no more. The next
> >> time the client tries to write to the socket, it gets into a really
> >> long TCP RST retry loop. To avoid this, we added a producer config
> >> that would reconnect to the VIP after n requests or m ms, whatever
> >> comes earlier.
> >>
> >> This config is no longer required in 0.8 and has been removed.
> >>
> >> Thanks,
> >> Neha
> >>
> >> On Sun, Nov 18, 2012 at 3:37 PM, Bae, Jae Hyeon <[EMAIL PROTECTED]>
> >> wrote:
> >> > Hi
> >> >
> >> > I am wondering that we do really need the following producer
> >> configuration
> >> >
> >> >   val reconnectInterval = Utils.getInt(props, "reconnect.interval",
> >> 30000)
> >> >
> >> >   /** negative reconnect time interval means disabling this time-based
> >> > reconnect feature */
> >> >   var reconnectTimeInterval = Utils.getInt(props,
> >> > "reconnect.time.interval.ms", 1000*1000*10)
> >> >
> >> > If these configurations have the history, please let me know.
> >> > Otherwise, I want to disable these default configurations.
> >> >
> >> > Thank you
> >> > Best, Jae
> >>
>