Kafka, mail # user - Re: Looks like consumer fetchers get stopped we are not getting any data - 2014-01-10, 23:37
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: Looks like consumer fetchers get stopped we are not getting any data
If a consumer rebalances for any reason (e.g., if a consumer
in the group has a soft failure such as a long GC) then the fetchers
are stopped as part of the rebalance process. The sequence is as
follows:

- Stop fetchers
- Commit offsets
- Release partition ownership
- Rebalance (i.e., figure out what partitions this consumer should now
  consume with the updated set of consumers)
- Acquire partition ownership
- Add fetchers to those partitions and resume consumption

i.e., rebalances should complete successfully and fetching should
resume. If you have any rebalance failures (search for "can't
rebalance after") then the consumer will effectively stop.

From later in this thread it seems your consumer somehow got into a
weird state in zookeeper, so your only recourse at this point may be
to stop all your consumers and restart.

Thanks,

Joel
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB