Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - How can a producer avoid slow brokers?


+
Raghu Angadi 2013-01-04, 15:53
+
Jun Rao 2013-01-04, 16:40
+
Raghu Angadi 2013-01-04, 17:17
Copy link to this message
-
Re: How can a producer avoid slow brokers?
Neha Narkhede 2013-01-04, 18:50
queue.enqueueTimeout.ms = -1 will block the producer instead of dropping
messages. This might be useful if you have the producer being fed by scribe
aggregators.

Thanks,
Neha
On Fri, Jan 4, 2013 at 9:04 AM, Raghu Angadi <[EMAIL PROTECTED]> wrote:

> On Fri, Jan 4, 2013 at 8:39 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Do you know why some of the brokers are much slower than others?
>
>
> We are currently running these in a shared environment, to make things
> worse these machines have single spindle. We have to put up with that until
> we move the brokers to dedicated hardware with multiple spindles. The
> problem is a bit exaggerated in current setup.
>
> Even with dedicated hardware, I am expecting some variation. One slightly
> degraded disk out of 12 could reduce effective b/w on all the spindles.
> Unfortunately there will be occasional rack level network slowdowns that
> take many hours to get fixed.
>
> In our case, we cannot let the back pressure from slow brokers propagate
> upstream. Producers receive messages from scribe aggregators and just have
> to drop the messages it they can't write fast enough.
>

 
+
Raghu Angadi 2013-01-04, 19:09
+
Jay Kreps 2013-01-04, 23:03