Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Questions about failure recovery, load balancing and producer ack/async mode


Copy link to this message
-
Re: Questions about failure recovery, load balancing and producer ack/async mode
Jun Rao 2013-10-24, 04:02
1,3 Take a look at request.required.acks in
http://kafka.apache.org/documentation.html#producerconfigs

2. The producer does random distribution by default. However, you can
provide a partitioning key and a partitioning function. For details on how
consumer load balancing works, see
http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf
Thanks,

Jun
On Wed, Oct 23, 2013 at 12:12 PM, Shafaq <[EMAIL PROTECTED]> wrote:

> Hi,
>
>    Following need more elaboration after reading kafka docs:
>
> 1- In a scenario during leader fails over, what happens to messages that
> are not committed to other followers and to the messages that producer keep
> in sending (in async mode) till new leader is elected. Can the producer
> buffer these messages till the failover gets completed (Do we see
> exceptions. Are these all lost?
>
> 2. Broker load distribution or to be exact partition load distribution is
> done via hashed key mod no of partition, is that correct?  How is it done
> at the consumer side where one thread per partition (for high-level
> consumer) How is the faliover handlled
>
> 3. How do we configure producer for wait for leader ack and wait for
> commited act which Jun  Rao talks about in his 2013 apachecon presentation
>
> --
> Kind Regards,
> Shafaq
>