Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> How can a producer avoid slow brokers?


+
Raghu Angadi 2013-01-04, 15:53
+
Jun Rao 2013-01-04, 16:40
+
Raghu Angadi 2013-01-04, 17:17
Copy link to this message
-
Re: How can a producer avoid slow brokers?
queue.enqueueTimeout.ms = -1 will block the producer instead of dropping
messages. This might be useful if you have the producer being fed by scribe
aggregators.

Thanks,
Neha
On Fri, Jan 4, 2013 at 9:04 AM, Raghu Angadi <[EMAIL PROTECTED]> wrote:

> On Fri, Jan 4, 2013 at 8:39 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Do you know why some of the brokers are much slower than others?
>
>
> We are currently running these in a shared environment, to make things
> worse these machines have single spindle. We have to put up with that until
> we move the brokers to dedicated hardware with multiple spindles. The
> problem is a bit exaggerated in current setup.
>
> Even with dedicated hardware, I am expecting some variation. One slightly
> degraded disk out of 12 could reduce effective b/w on all the spindles.
> Unfortunately there will be occasional rack level network slowdowns that
> take many hours to get fixed.
>
> In our case, we cannot let the back pressure from slow brokers propagate
> upstream. Producers receive messages from scribe aggregators and just have
> to drop the messages it they can't write fast enough.
>

 
+
Raghu Angadi 2013-01-04, 19:09
+
Jay Kreps 2013-01-04, 23:03
+
Raghu Angadi 2013-01-09, 20:06
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB