Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - How can a producer avoid slow brokers?


+
Raghu Angadi 2013-01-04, 15:53
Copy link to this message
-
Re: How can a producer avoid slow brokers?
Jun Rao 2013-01-04, 16:40
Do you know why some of the brokers are much slower than others?

Thanks,

Jun

On Fri, Jan 4, 2013 at 12:24 AM, Raghu Angadi <[EMAIL PROTECTED]> wrote:

> Producer distributes messages uniformly across the partitions.
>
> This does not work very well when some of the brokers are much slower than
> others. Is there a way to temporarily avoid such slow brokers?
>
> While async producers, I could avoid producers that have lot more messages
> in their internal queue compared to others (through my own Partitioners).
> But the queue size is not available. tried to maintain my own estimate of
> queue size using 'CallbackHandler', but API does not seem to provide enough
> info (it provides partition id, but not broker id. plus, CallbackHandler
> seems to be removed in 0.8).
>
> any suggestions?
>
> Kafka version : 0.7.1
>
> thanks,
> Raghu.
>

 
+
Raghu Angadi 2013-01-04, 17:17
+
Neha Narkhede 2013-01-04, 18:50
+
Raghu Angadi 2013-01-04, 19:09
+
Jay Kreps 2013-01-04, 23:03