Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - 0.72 Kafka - ZK over VPN


Copy link to this message
-
Re: 0.72 Kafka - ZK over VPN
Philip O'Toole 2013-11-28, 05:12
Great Jun, thanks. As always, we've found Kafka 0.72 to be rock solid under
lots of different conditions and failures -- except when the disks fill up,
but I can't blame it for that. :-)

Philip
On Wed, Nov 27, 2013 at 8:44 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Both the broker and the consumer have the logic for handling ZK session
> expirations. So, they should recover automatically. The issue is that if
> there is a real failure in the broker/consumer while the VPN is down, the
> failure may not be detected.
>
> Thanks,
>
> Jun
>
>
> On Wed, Nov 27, 2013 at 8:02 AM, Philip O'Toole <[EMAIL PROTECTED]> wrote:
>
> > Actually, we are using custom producers, which recover fully after any
> > disconnect-and-reconnect from Kafka. It is the High-Level consumers and
> > Kafka itself than concerned me.
> >
> > I have seen good behaviour from the system in these conditions before,
> but
> > wanted to confirm.
> >
> > Philip
> >
> >
> >
> >
> > On Wed, Nov 27, 2013 at 7:46 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > I assume that you are using a ZK-based producer. If brokers don't
> change
> > in
> > > the window where the VPN connection is down, this may not matter. When
> > the
> > > VPN connection is back, the ZK session will expire and the producer
> will
> > > establish establish a new ZK session and  new connections to the
> brokers.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Nov 26, 2013 at 9:33 PM, Philip O'Toole <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > I want to use a ZK cluster for my Kafka cluster, which is only
> > available
> > > > over a cross-country VPN tunnel. The VPN tunnel is prone to resets,
> > every
> > > > other day or so, perhaps down for a couple of  minutes at a time.
> > > >
> > > > Is this a concern? Any setting changes I should make to mitigate any
> > > > potential issues? Will my simple producers be affected at all, while
> > > > writing to a Kafka broker that experiences this?
> > > >
> > > > Thanks,
> > > >
> > > > Philip
> > >
> >
>