Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> How can a producer avoid slow brokers?

Copy link to this message
Re: How can a producer avoid slow brokers?
On Fri, Jan 4, 2013 at 8:39 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Do you know why some of the brokers are much slower than others?
We are currently running these in a shared environment, to make things
worse these machines have single spindle. We have to put up with that until
we move the brokers to dedicated hardware with multiple spindles. The
problem is a bit exaggerated in current setup.

Even with dedicated hardware, I am expecting some variation. One slightly
degraded disk out of 12 could reduce effective b/w on all the spindles.
Unfortunately there will be occasional rack level network slowdowns that
take many hours to get fixed.

In our case, we cannot let the back pressure from slow brokers propagate
upstream. Producers receive messages from scribe aggregators and just have
to drop the messages it they can't write fast enough.