Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - [0.6] Consumer API Question - What if a ConsumerIterator has an error?


Copy link to this message
-
[0.6] Consumer API Question - What if a ConsumerIterator has an error?
David Ross 2011-12-22, 09:40
Howdy,

I have come across a potential issue with the Consumer API in Kafka 0.6.
After some tests and reading through the code, I believe the following
scenario could result in an unhappy situation:

   1. Create 5 streams listening to a topic
   2. One of the streams times out (or has some other error) and it's
   iterator is now in mode FAILED
   3. (I think) Messages will now come into all five streams, but one of
   them won't yield any new messages because of (2)

The only solution I can think of is if an error is detected in any of the
streams, all of them should be stopped and a new set of streams should be
made. This seems unfavorable and would require messy concurrent programming.

So is this an actual problem? Is there a better workaround? Have I
misinterpreted something?
Thanks,

David