Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> ASyncProducer loss more messages than SyncProducer when broker crashes.


+
王国栋 2013-02-04, 06:01
+
Jun Rao 2013-02-04, 16:44
+
王国栋 2013-02-05, 03:12
+
Jun Rao 2013-02-05, 06:20
+
王国栋 2013-02-05, 06:31
+
Jun Rao 2013-02-05, 09:03
Copy link to this message
-
Re: ASyncProducer loss more messages than SyncProducer when broker crashes.
Thanks Jun.
On Tue, Feb 5, 2013 at 5:03 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> It could be. This is because it takes ZK up to 6 secs to detect the failure
> due to a hard kill (since ZK client is not closed properly). You can shrink
> ZK session timeout to reduce the delay.
>
> Thanks,
>
> Jun
>
> On Mon, Feb 4, 2013 at 10:30 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
>
> > We kill the broker on purpose, since we are doing some fail over tests.
> If
> > the broker is hard killed, will the async-producer lose message for 6
> > seconds?
> >
> > Thanks.
> >
> > Guodong
> >
> >
> > On Tue, Feb 5, 2013 at 2:19 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Is the broker shut down cleanly? If so, the producer should get a ZK
> > > notification immediately. If the broker is hard killed, the ZK
> > notification
> > > won't arrive at the producer until zk session timeout (default is 6
> > secs).
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Feb 4, 2013 at 7:11 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
> > >
> > > > Hi Jun
> > > >
> > > > We are using zookeeper. And the async-producer loses  messages in 'X'
> > > > seconds. Here, X is between 4 and 6. It seems that X is comparable
> with
> > > > zookeeper session timeout.
> > > >
> > > > I am wondering the behavior of async-producer when socket is broken,
> > but
> > > > the broker is still visible for zookeeper client.
> > > >
> > > >
> > > > On Tue, Feb 5, 2013 at 12:39 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Currently, there is no callback to report socket failure in the
> async
> > > > > producer. However, async producer should be able to detect socket
> > > failure
> > > > > and reconnect to another broker. Are you using a vip or Zookeeper
> in
> > > the
> > > > > producer for load balancing?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Sun, Feb 3, 2013 at 10:00 PM, 王国栋 <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > we are doing some fail over test on Kafka0.7.2. We kill one of
> the
> > > > > brokers
> > > > > > on purpose. But we find that the async-producer loss more
> messages.
> > > We
> > > > do
> > > > > > catch exceptions for send api. But it seems that the
> async-producer
> > > > > doesn't
> > > > > > throw any exceptions when broker is done.
> > > > > >
> > > > > > We change the producer to sync-producer, and got the exceptions
> as
> > > soon
> > > > > as
> > > > > > the broker is down.
> > > > > >
> > > > > > So I am wondering if I can get the socket status connected to the
> > > > broker.
> > > > > > Does some callbacks can do this?
> > > > > >
> > > > > > Thanks.
> > > > > > --
> > > > > > Guodong Wang
> > > > > > 王国栋
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Guodong Wang
> > > > 王国栋
> > > >
> > >
> >
> >
> >
> > --
> > Guodong Wang
> > 王国栋
> >
>

--
Guodong Wang
王国栋

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB