Kafka, mail # user - Re: Consumer throughput imbalance - 2013-08-25, 17:24
 Search Hadoop and all its subprojects:

Switch to Plain View
+
Ian Friedman 2013-08-24, 16:59
+
Neha Narkhede 2013-08-25, 14:46
+
Ian Friedman 2013-08-25, 16:01
+
Mark 2013-08-25, 16:14
+
Ian Friedman 2013-08-25, 17:17
Copy link to this message
-
Re: Consumer throughput imbalance
Sorry I reread what I've written so far and found that it doesn't state the actual problem very well. Let me clarify once again:

The problem we're trying to solve is that we can't let messages go for unbounded amounts of time without getting processed, and it seems that something about what we're doing (which I suspect is the fact that consumers own several partitions but only consume from one of them at a time until it's caught up) is causing a small number of them to sit around for hours and hours. This is despite some consumers idling due to being fully caught up on the partitions they own. We've found that requeueing the oldest messages (consumers ignore messages that have already been processed) is fairly effective in getting them to go away, but I'm looking for a more stable solution.

Ian Friedman
On Sunday, August 25, 2013 at 1:15 PM, Ian Friedman wrote:

 
+
Jay Kreps 2013-08-25, 19:12
+
Ian Friedman 2013-08-26, 05:02
+
Jay Kreps 2013-08-26, 17:27
+
Ian Friedman 2013-08-26, 18:19
+
Ian Friedman 2013-08-26, 18:34
+
Jay Kreps 2013-08-26, 18:38
+
Ian Friedman 2013-08-26, 21:43
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB