Hi,

we are doing some fail over test on Kafka0.7.2. We kill one of the brokers
on purpose. But we find that the async-producer loss more messages. We do
catch exceptions for send api. But it seems that the async-producer doesn't
throw any exceptions when broker is done.

We change the producer to sync-producer, and got the exceptions as soon as
the broker is down.

So I am wondering if I can get the socket status connected to the broker.
Does some callbacks can do this?

Thanks.
--
Guodong Wang
王国栋

 

Currently, there is no callback to report socket failure in the async
producer. However, async producer should be able to detect socket failure
and reconnect to another broker. Are you using a vip or Zookeeper in the
producer for load balancing?

Thanks,

Jun

On Sun, Feb 3, 2013 at 10:00 PM, Íõ¹ú¶° <[EMAIL PROTECTED]> wrote:

> Hi,
>
> we are doing some fail over test on Kafka0.7.2. We kill one of the brokers
> on purpose. But we find that the async-producer loss more messages. We do
> catch exceptions for send api. But it seems that the async-producer doesn't
> throw any exceptions when broker is done.
>
> We change the producer to sync-producer, and got the exceptions as soon as
> the broker is down.
>
> So I am wondering if I can get the socket status connected to the broker.
> Does some callbacks can do this?
>
> Thanks.
> --
> Guodong Wang
> Íõ¹ú¶°
>

 

Hi Jun

We are using zookeeper. And the async-producer loses  messages in 'X'
seconds. Here, X is between 4 and 6. It seems that X is comparable with
zookeeper session timeout.

I am wondering the behavior of async-producer when socket is broken, but
the broker is still visible for zookeeper client.
On Tue, Feb 5, 2013 at 12:39 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Currently, there is no callback to report socket failure in the async
> producer. However, async producer should be able to detect socket failure
> and reconnect to another broker. Are you using a vip or Zookeeper in the
> producer for load balancing?
>
> Thanks,
>
> Jun
>
> On Sun, Feb 3, 2013 at 10:00 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
>
> > Hi,
> >
> > we are doing some fail over test on Kafka0.7.2. We kill one of the
> brokers
> > on purpose. But we find that the async-producer loss more messages. We do
> > catch exceptions for send api. But it seems that the async-producer
> doesn't
> > throw any exceptions when broker is done.
> >
> > We change the producer to sync-producer, and got the exceptions as soon
> as
> > the broker is down.
> >
> > So I am wondering if I can get the socket status connected to the broker.
> > Does some callbacks can do this?
> >
> > Thanks.
> > --
> > Guodong Wang
> > 王国栋
> >
>

--
Guodong Wang
王国栋

 

Is the broker shut down cleanly? If so, the producer should get a ZK
notification immediately. If the broker is hard killed, the ZK notification
won't arrive at the producer until zk session timeout (default is 6 secs).

Thanks,

Jun

On Mon, Feb 4, 2013 at 7:11 PM, 王国栋 <[EMAIL PROTECTED]> wrote:

> Hi Jun
>
> We are using zookeeper. And the async-producer loses  messages in 'X'
> seconds. Here, X is between 4 and 6. It seems that X is comparable with
> zookeeper session timeout.
>
> I am wondering the behavior of async-producer when socket is broken, but
> the broker is still visible for zookeeper client.
>
>
> On Tue, Feb 5, 2013 at 12:39 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Currently, there is no callback to report socket failure in the async
> > producer. However, async producer should be able to detect socket failure
> > and reconnect to another broker. Are you using a vip or Zookeeper in the
> > producer for load balancing?
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Feb 3, 2013 at 10:00 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
> >
> > > Hi,
> > >
> > > we are doing some fail over test on Kafka0.7.2. We kill one of the
> > brokers
> > > on purpose. But we find that the async-producer loss more messages. We
> do
> > > catch exceptions for send api. But it seems that the async-producer
> > doesn't
> > > throw any exceptions when broker is done.
> > >
> > > We change the producer to sync-producer, and got the exceptions as soon
> > as
> > > the broker is down.
> > >
> > > So I am wondering if I can get the socket status connected to the
> broker.
> > > Does some callbacks can do this?
> > >
> > > Thanks.
> > > --
> > > Guodong Wang
> > > 王国栋
> > >
> >
>
>
>
> --
> Guodong Wang
> 王国栋
>

 

We kill the broker on purpose, since we are doing some fail over tests. If
the broker is hard killed, will the async-producer lose message for 6
seconds?

Thanks.

Guodong
On Tue, Feb 5, 2013 at 2:19 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Is the broker shut down cleanly? If so, the producer should get a ZK
> notification immediately. If the broker is hard killed, the ZK notification
> won't arrive at the producer until zk session timeout (default is 6 secs).
>
> Thanks,
>
> Jun
>
> On Mon, Feb 4, 2013 at 7:11 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
>
> > Hi Jun
> >
> > We are using zookeeper. And the async-producer loses  messages in 'X'
> > seconds. Here, X is between 4 and 6. It seems that X is comparable with
> > zookeeper session timeout.
> >
> > I am wondering the behavior of async-producer when socket is broken, but
> > the broker is still visible for zookeeper client.
> >
> >
> > On Tue, Feb 5, 2013 at 12:39 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Currently, there is no callback to report socket failure in the async
> > > producer. However, async producer should be able to detect socket
> failure
> > > and reconnect to another broker. Are you using a vip or Zookeeper in
> the
> > > producer for load balancing?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sun, Feb 3, 2013 at 10:00 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
> > >
> > > > Hi,
> > > >
> > > > we are doing some fail over test on Kafka0.7.2. We kill one of the
> > > brokers
> > > > on purpose. But we find that the async-producer loss more messages.
> We
> > do
> > > > catch exceptions for send api. But it seems that the async-producer
> > > doesn't
> > > > throw any exceptions when broker is done.
> > > >
> > > > We change the producer to sync-producer, and got the exceptions as
> soon
> > > as
> > > > the broker is down.
> > > >
> > > > So I am wondering if I can get the socket status connected to the
> > broker.
> > > > Does some callbacks can do this?
> > > >
> > > > Thanks.
> > > > --
> > > > Guodong Wang
> > > > 王国栋
> > > >
> > >
> >
> >
> >
> > --
> > Guodong Wang
> > 王国栋
> >
>

--
Guodong Wang
王国栋

 

It could be. This is because it takes ZK up to 6 secs to detect the failure
due to a hard kill (since ZK client is not closed properly). You can shrink
ZK session timeout to reduce the delay.

Thanks,

Jun

On Mon, Feb 4, 2013 at 10:30 PM, 王国栋 <[EMAIL PROTECTED]> wrote:

> We kill the broker on purpose, since we are doing some fail over tests. If
> the broker is hard killed, will the async-producer lose message for 6
> seconds?
>
> Thanks.
>
> Guodong
>
>
> On Tue, Feb 5, 2013 at 2:19 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Is the broker shut down cleanly? If so, the producer should get a ZK
> > notification immediately. If the broker is hard killed, the ZK
> notification
> > won't arrive at the producer until zk session timeout (default is 6
> secs).
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Feb 4, 2013 at 7:11 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
> >
> > > Hi Jun
> > >
> > > We are using zookeeper. And the async-producer loses  messages in 'X'
> > > seconds. Here, X is between 4 and 6. It seems that X is comparable with
> > > zookeeper session timeout.
> > >
> > > I am wondering the behavior of async-producer when socket is broken,
> but
> > > the broker is still visible for zookeeper client.
> > >
> > >
> > > On Tue, Feb 5, 2013 at 12:39 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > > > Currently, there is no callback to report socket failure in the async
> > > > producer. However, async producer should be able to detect socket
> > failure
> > > > and reconnect to another broker. Are you using a vip or Zookeeper in
> > the
> > > > producer for load balancing?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Sun, Feb 3, 2013 at 10:00 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > we are doing some fail over test on Kafka0.7.2. We kill one of the
> > > > brokers
> > > > > on purpose. But we find that the async-producer loss more messages.
> > We
> > > do
> > > > > catch exceptions for send api. But it seems that the async-producer
> > > > doesn't
> > > > > throw any exceptions when broker is done.
> > > > >
> > > > > We change the producer to sync-producer, and got the exceptions as
> > soon
> > > > as
> > > > > the broker is down.
> > > > >
> > > > > So I am wondering if I can get the socket status connected to the
> > > broker.
> > > > > Does some callbacks can do this?
> > > > >
> > > > > Thanks.
> > > > > --
> > > > > Guodong Wang
> > > > > 王国栋
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Guodong Wang
> > > 王国栋
> > >
> >
>
>
>
> --
> Guodong Wang
> 王国栋
>

 

Thanks Jun.
On Tue, Feb 5, 2013 at 5:03 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> It could be. This is because it takes ZK up to 6 secs to detect the failure
> due to a hard kill (since ZK client is not closed properly). You can shrink
> ZK session timeout to reduce the delay.
>
> Thanks,
>
> Jun
>
> On Mon, Feb 4, 2013 at 10:30 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
>
> > We kill the broker on purpose, since we are doing some fail over tests.
> If
> > the broker is hard killed, will the async-producer lose message for 6
> > seconds?
> >
> > Thanks.
> >
> > Guodong
> >
> >
> > On Tue, Feb 5, 2013 at 2:19 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Is the broker shut down cleanly? If so, the producer should get a ZK
> > > notification immediately. If the broker is hard killed, the ZK
> > notification
> > > won't arrive at the producer until zk session timeout (default is 6
> > secs).
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Feb 4, 2013 at 7:11 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
> > >
> > > > Hi Jun
> > > >
> > > > We are using zookeeper. And the async-producer loses  messages in 'X'
> > > > seconds. Here, X is between 4 and 6. It seems that X is comparable
> with
> > > > zookeeper session timeout.
> > > >
> > > > I am wondering the behavior of async-producer when socket is broken,
> > but
> > > > the broker is still visible for zookeeper client.
> > > >
> > > >
> > > > On Tue, Feb 5, 2013 at 12:39 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Currently, there is no callback to report socket failure in the
> async
> > > > > producer. However, async producer should be able to detect socket
> > > failure
> > > > > and reconnect to another broker. Are you using a vip or Zookeeper
> in
> > > the
> > > > > producer for load balancing?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Sun, Feb 3, 2013 at 10:00 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > we are doing some fail over test on Kafka0.7.2. We kill one of
> the
> > > > > brokers
> > > > > > on purpose. But we find that the async-producer loss more
> messages.
> > > We
> > > > do
> > > > > > catch exceptions for send api. But it seems that the
> async-producer
> > > > > doesn't
> > > > > > throw any exceptions when broker is done.
> > > > > >
> > > > > > We change the producer to sync-producer, and got the exceptions
> as
> > > soon
> > > > > as
> > > > > > the broker is down.
> > > > > >
> > > > > > So I am wondering if I can get the socket status connected to the
> > > > broker.
> > > > > > Does some callbacks can do this?
> > > > > >
> > > > > > Thanks.
> > > > > > --
> > > > > > Guodong Wang
> > > > > > 王国栋
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Guodong Wang
> > > > 王国栋
> > > >
> > >
> >
> >
> >
> > --
> > Guodong Wang
> > 王国栋
> >
>

--
Guodong Wang
王国栋