Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> How can a producer avoid slow brokers?

Raghu Angadi 2013-01-04, 15:53
Jun Rao 2013-01-04, 16:40
Raghu Angadi 2013-01-04, 17:17
Copy link to this message
Re: How can a producer avoid slow brokers?
queue.enqueueTimeout.ms = -1 will block the producer instead of dropping
messages. This might be useful if you have the producer being fed by scribe

On Fri, Jan 4, 2013 at 9:04 AM, Raghu Angadi <[EMAIL PROTECTED]> wrote:

> On Fri, Jan 4, 2013 at 8:39 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > Do you know why some of the brokers are much slower than others?
> We are currently running these in a shared environment, to make things
> worse these machines have single spindle. We have to put up with that until
> we move the brokers to dedicated hardware with multiple spindles. The
> problem is a bit exaggerated in current setup.
> Even with dedicated hardware, I am expecting some variation. One slightly
> degraded disk out of 12 could reduce effective b/w on all the spindles.
> Unfortunately there will be occasional rack level network slowdowns that
> take many hours to get fixed.
> In our case, we cannot let the back pressure from slow brokers propagate
> upstream. Producers receive messages from scribe aggregators and just have
> to drop the messages it they can't write fast enough.

Raghu Angadi 2013-01-04, 19:09
Jay Kreps 2013-01-04, 23:03