Search Hadoop and all its sub project:

Switch to Threaded View
Subject: Re: Kafka for cluster fan out messaging
consumer mapping is client side logic to the high level consumer, yes.

partition -> client mapping is automatically handled by the high level
consumer.  this is organized by groupId

when groupId is the same the consumers operate together and consumers
within the group do not see the same information.  when groupId is
different then they will see the same information and manage their offsets

the kafka high level consumer will rebalance consumers for partitions
available.  if every consumer has at most one partition and there are more
consumers in the group still then those you should treat like consumer side
fail over.   so you can have twice the number of consumers you have
partitions and lose 50% of your infrastructure and not lose any performance
through the pipeline.

 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 Twitter: @allthingshadoop <>
On Sun, Dec 29, 2013 at 3:18 PM, Jay Beavers <[EMAIL PROTECTED]>wrote:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB