Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> async producer behavior if zk and/or kafka cluster goes away...


Copy link to this message
-
async producer behavior if zk and/or kafka cluster goes away...
I've been doing some testing, with an async producer.

It seems, if I start up the producer, with no zk cluster present, it does
what I expect, that is it waits for a limited time looking for the zk
cluster, and then gives up after the zk.connectiontimeout.ms setting
(6000ms, by default), and fails to send a message.  However, if after
starting up zk and having a good connection to zk and kafka, I then
shutdown the zk cluster, the producer never seems to stop accepting
messages to send.

As long as kafka stays up and running, even without zk still available, my
producer sends messages and my consumer can consume them.

However, if I then stop kafka also, my producer happily keeps on accepting
messages without failing in a call to producer.send().  It's clearly no
longer able to send any messages at this point.  So, I assume it eventually
will just start dropping messages on the floor?

I would have expected that once both zk/kafka are not available, things
should revert to the initial startup case, where it tries for 6000ms and
then throws an exception on send.

Thoughts?

What's the expected behavior for async producers, when the async buffered
messages can't be sent.  I think it's fine if they are just lost, but
should it be possible to block further accepting of messages once the
system has detected a problem communicating with zk/kafka?

Also, if I cleanly shutdown an async producer (e.g. call producer.close()),
should it make a best effort to send out any buffered messages before
shutting down?  Or will it quit immediately dropping any buffered messages
on the floor?

Jason
+
Jason Rosenberg 2012-11-19, 21:27
+
Jay Kreps 2012-11-19, 22:31
+
Jason Rosenberg 2012-11-19, 23:30
+
Jay Kreps 2012-11-20, 04:50
+
Jason Rosenberg 2012-11-20, 15:57
+
Neha Narkhede 2012-11-20, 16:41
+
Jun Rao 2012-11-20, 17:03
+
Jason Rosenberg 2012-11-20, 17:44
+
Neha Narkhede 2012-11-20, 18:00
+
Jason Rosenberg 2012-11-20, 18:04
+
Jay Kreps 2012-11-20, 18:20
+
Jason Rosenberg 2012-11-20, 18:28
+
Neha Narkhede 2012-11-20, 18:35
+
Jun Rao 2012-11-20, 18:39
+
Jason Rosenberg 2012-11-20, 19:09
+
Neha Narkhede 2012-11-20, 19:18
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB