Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> SimpleConsumer error conditions and handling


Copy link to this message
-
SimpleConsumer error conditions and handling
Hi,

0.8.0 HEAD from 3/4/2013.

As I think through building a robust SimpleConsumer I ran some failure
tests today and want to make sure I understand what is going on.

FYI I know that I should be doing a metadata lookup to find the leader, but
I wanted to see what happens if things are going well and the leader
changes between requests or I've cached the leader and try to connect
without the cost of a leader lookup.

First test: connect to a Broker that is a 'copy' of the topic/partition but
not leader. Get an error '5' which maps to
'ErrorMapping.LeaderNotAvailableCode'.

Why didn't I get ErrorMapping.NotLeaderForPartitionCode or something else
to tell me I'm not talking to the Leader? 'not available' implies something
is wrong with replication. But connecting to the leader Broker everything
works fine.

Second test: connect to a Broker that isn't the leader or a copy and I get
error 3, unknown topic or partition. Makes sense.

Third test: connect to the leader and while reading data, shutdown the
leader Broker via command line: I get some IOExceptions then Connection
Refused on the reconnect. (Note that the Connect Refused is the exception
raised, IOException was written to logs but not raised to my code.)

Not sure the best way to code to recover from this without assuming the
worst every time  Could there be some notice from Kafka that the connection
to the leader was closed due to a shutdown vs. getting Connection Refused
errors so I can respond differently? Something like 'Broker has closed
connection due to shutdown'. So I know to sleep for a second before going
through the leader lookup logic again? Or ideally have Kafka know it was a
clean shutdown and automatically transition to the new leader.

Knowing it was a clean shutdown would also allow me to treat the clean
shutdown as a normal occurrence vs. an exception when something goes wrong.

Thanks,

Chris

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB