Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> How can a producer avoid slow brokers?


Copy link to this message
-
How can a producer avoid slow brokers?
Producer distributes messages uniformly across the partitions.

This does not work very well when some of the brokers are much slower than
others. Is there a way to temporarily avoid such slow brokers?

While async producers, I could avoid producers that have lot more messages
in their internal queue compared to others (through my own Partitioners).
But the queue size is not available. tried to maintain my own estimate of
queue size using 'CallbackHandler', but API does not seem to provide enough
info (it provides partition id, but not broker id. plus, CallbackHandler
seems to be removed in 0.8).

any suggestions?

Kafka version : 0.7.1

thanks,
Raghu.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB