Kafka, mail # user - Re: High level consumer Blocked when there is still message in topic - 2013-11-13, 21:23
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Plain View
+
李帅 2013-11-13, 04:42
+
Jun Rao 2013-11-13, 05:28
+
hsy541@... 2013-11-13, 19:54
Copy link to this message
-
Re: High level consumer Blocked when there is still message in topic
On Wed, Nov 13, 2013 at 11:54:07AM -0800, [EMAIL PROTECTED] wrote:

Depending on your CPU-specs you should be able to add threads to scale
out - but yes if you want to scale out even more you would want more
instances on more nodes provided there are sufficient partitions to
load balance.
The high-level consumer has a heavy dependence on zookeeper - so yes
there is a bottleneck there especially if each consumer consumes a lot
of topics. This will be addressed to a large degree by the client
rewrite; consumer coordinator approach
(https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPI)
and in-built offset management
(https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management)

Joel
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB