Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Re: how brokers sync with each other


Copy link to this message
-
Re: how brokers sync with each other
Joel Koshy 2013-06-29, 01:04
I'm assuming this is somewhat related to your previous question on
cross-DC replication. This is not an ideal set up as mentioned there.
If the replica lags then it will fall out of the "in-sync-replica"
set. You could tune parameters that effectively allow a high (but
bounded over time) lag between the replica and the leader. There is no
buffer per se - the replicas just keep fetching from the leader (which
would have appended the messages to its log).

If your producer has acks set to -1 (which means wait until all
replicas in ISR have received the message) it would slow down your
producer considerably. (You could use acks = 1 in that case though but
with weaker guarantees - i.e., your message could be lost in failure
scenarios).

Joel
On Fri, Jun 28, 2013 at 5:41 PM, Yu, Libo <[EMAIL PROTECTED]> wrote:
> Hi,
>
> Assume A and B are two brokers in a kafka server. And there is long network
> Latency between A and B. For a partition with two replications, one replication
> Is assigned to A and the other is assigned to B. Number of acknowledge is set
> to one. Assume the partition is handled by broker A.
>
> After a message is accepted and saved by A, ack will be sent and the message
> will be sent to broker B at the same time. But due to long network latency between
> A and B, it is possible that B lags behind A. In this case, is there a buffer on broker A
> that holds all messages to be written to B? Is there any limit on the lag between A
> and B?
>
> Compare with the case when there is no network latency between A and B, any
> difference in publishing speed?
>
> Thanks
>
> Libo
>