Kafka, mail # user - Re: consumer partition rebalancing - 2013-09-11, 17:09
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Re: consumer partition rebalancing
Thanks Joel. Just to be sure - SimpleConsumer (or an IntermediateConsumer as suggested below) does not join in partition rebalancing when another consumer in it's group joins or a new broker joins or a new topic is created?
________________________________
 From: Joel Koshy <[EMAIL PROTECTED]>
To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; Kam Kasravi <[EMAIL PROTECTED]>
Sent: Tuesday, September 10, 2013 4:36 PM
Subject: Re: consumer partition rebalancing
 
To some degree - by either reducing the number of streams that C(i) is
configured with; or by removing it altogether.
Right now, no. I think what you may be effectively suggesting is to
support some form of pluggable logic for partition ownership which
should work as long as it is consistent across all consumer instances.
However, a designated assignment list could be achieved by using
SimpleConsumer - although you would then need to write a little code
to react to leader changes. (We currently don't have a
intermediate-level consumer - i.e., a consumer that can be designated
to consume from a given partition and automatically react to leader
changes.)

Joel
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB