Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> async producer behavior if zk and/or kafka cluster goes away...

Copy link to this message
Re: async producer behavior if zk and/or kafka cluster goes away...
I forgot to mention, that I'm working with a recent version of the 0.8 code
(Last chaned rev: 1396425).


On Mon, Nov 19, 2012 at 1:23 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> I've been doing some testing, with an async producer.
> It seems, if I start up the producer, with no zk cluster present, it does
> what I expect, that is it waits for a limited time looking for the zk
> cluster, and then gives up after the zk.connectiontimeout.ms setting
> (6000ms, by default), and fails to send a message.  However, if after
> starting up zk and having a good connection to zk and kafka, I then
> shutdown the zk cluster, the producer never seems to stop accepting
> messages to send.
> As long as kafka stays up and running, even without zk still available, my
> producer sends messages and my consumer can consume them.
> However, if I then stop kafka also, my producer happily keeps on accepting
> messages without failing in a call to producer.send().  It's clearly no
> longer able to send any messages at this point.  So, I assume it eventually
> will just start dropping messages on the floor?
> I would have expected that once both zk/kafka are not available, things
> should revert to the initial startup case, where it tries for 6000ms and
> then throws an exception on send.
> Thoughts?
> What's the expected behavior for async producers, when the async buffered
> messages can't be sent.  I think it's fine if they are just lost, but
> should it be possible to block further accepting of messages once the
> system has detected a problem communicating with zk/kafka?
> Also, if I cleanly shutdown an async producer (e.g. call
> producer.close()), should it make a best effort to send out any buffered
> messages before shutting down?  Or will it quit immediately dropping any
> buffered messages on the floor?
> Jason