Kafka, mail # user - Re: How can a producer avoid slow brokers? - 2013-01-04, 19:09
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Plain View
+
Raghu Angadi 2013-01-04, 15:53
+
Jun Rao 2013-01-04, 16:40
+
Raghu Angadi 2013-01-04, 17:17
+
Neha Narkhede 2013-01-04, 18:50
Copy link to this message
-
Re: How can a producer avoid slow brokers?
We can't afford to block since we don't want the pressure
to percolate upstream (even if we did, scribe would drop the messages
rather than the producer).

aim is to not block while queueing as long as there are enough brokers that
can handle the load.

with infinite timeout (or even with smaller time out like 1 second), when a
thread invokes producer.send(ProducerData) it does not know whether it is
going to block or not. I am trying to write a partitioner that can avoid
asyncProducers with higher backlog their queue.

Of course, if the brokers in aggregate can't handle that load, then there
is no choice but to drop the messages. But currently we end up dropping
messages even with one single slow broker.

Raghu.
On Fri, Jan 4, 2013 at 10:49 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
 
+
Jay Kreps 2013-01-04, 23:03
+
Raghu Angadi 2013-01-09, 20:06
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB