Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> async producer behavior if zk and/or kafka cluster goes away...


Copy link to this message
-
Re: async producer behavior if zk and/or kafka cluster goes away...
I forgot to mention, that I'm working with a recent version of the 0.8 code
(Last chaned rev: 1396425).

Jason

On Mon, Nov 19, 2012 at 1:23 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> I've been doing some testing, with an async producer.
>
> It seems, if I start up the producer, with no zk cluster present, it does
> what I expect, that is it waits for a limited time looking for the zk
> cluster, and then gives up after the zk.connectiontimeout.ms setting
> (6000ms, by default), and fails to send a message.  However, if after
> starting up zk and having a good connection to zk and kafka, I then
> shutdown the zk cluster, the producer never seems to stop accepting
> messages to send.
>
> As long as kafka stays up and running, even without zk still available, my
> producer sends messages and my consumer can consume them.
>
> However, if I then stop kafka also, my producer happily keeps on accepting
> messages without failing in a call to producer.send().  It's clearly no
> longer able to send any messages at this point.  So, I assume it eventually
> will just start dropping messages on the floor?
>
> I would have expected that once both zk/kafka are not available, things
> should revert to the initial startup case, where it tries for 6000ms and
> then throws an exception on send.
>
> Thoughts?
>
> What's the expected behavior for async producers, when the async buffered
> messages can't be sent.  I think it's fine if they are just lost, but
> should it be possible to block further accepting of messages once the
> system has detected a problem communicating with zk/kafka?
>
> Also, if I cleanly shutdown an async producer (e.g. call
> producer.close()), should it make a best effort to send out any buffered
> messages before shutting down?  Or will it quit immediately dropping any
> buffered messages on the floor?
>
> Jason
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB