Kafka, mail # user - Re: Consumer throughput imbalance - 2013-08-25, 14:46
 Search Hadoop and all its subprojects:

Switch to Plain View
+
Ian Friedman 2013-08-24, 16:59
Copy link to this message
-
Re: Consumer throughput imbalance
Making producer side partitioning depend on consumer behavior might not be
such a good idea. If consumption is a bottleneck, changing producer side
partitioning may not help. To relieve consumption bottleneck, you may need
to increase the number of partitions for those topics and increase the
number of consumer instances.

You mentioned that the consumers take longer to process certain kinds of
messages. What you can do is place the messages that require slower
processing in separate topics, so that you can scale the number of
partitions and number of consumer instances, for those messages
independently.

Thanks,
Neha
On Sat, Aug 24, 2013 at 9:57 AM, Ian Friedman <[EMAIL PROTECTED]> wrote:
 
+
Ian Friedman 2013-08-25, 16:01
+
Mark 2013-08-25, 16:14
+
Ian Friedman 2013-08-25, 17:17
+
Ian Friedman 2013-08-25, 17:24
+
Jay Kreps 2013-08-25, 19:12
+
Ian Friedman 2013-08-26, 05:02
+
Jay Kreps 2013-08-26, 17:27
+
Ian Friedman 2013-08-26, 18:19
+
Ian Friedman 2013-08-26, 18:34
+
Jay Kreps 2013-08-26, 18:38
+
Ian Friedman 2013-08-26, 21:43
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB