Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Re: how brokers sync with each other


Copy link to this message
-
Re: how brokers sync with each other
I'm assuming this is somewhat related to your previous question on
cross-DC replication. This is not an ideal set up as mentioned there.
If the replica lags then it will fall out of the "in-sync-replica"
set. You could tune parameters that effectively allow a high (but
bounded over time) lag between the replica and the leader. There is no
buffer per se - the replicas just keep fetching from the leader (which
would have appended the messages to its log).

If your producer has acks set to -1 (which means wait until all
replicas in ISR have received the message) it would slow down your
producer considerably. (You could use acks = 1 in that case though but
with weaker guarantees - i.e., your message could be lost in failure
scenarios).

Joel
On Fri, Jun 28, 2013 at 5:41 PM, Yu, Libo <[EMAIL PROTECTED]> wrote:
> Hi,
>
> Assume A and B are two brokers in a kafka server. And there is long network
> Latency between A and B. For a partition with two replications, one replication
> Is assigned to A and the other is assigned to B. Number of acknowledge is set
> to one. Assume the partition is handled by broker A.
>
> After a message is accepted and saved by A, ack will be sent and the message
> will be sent to broker B at the same time. But due to long network latency between
> A and B, it is possible that B lags behind A. In this case, is there a buffer on broker A
> that holds all messages to be written to B? Is there any limit on the lag between A
> and B?
>
> Compare with the case when there is no network latency between A and B, any
> difference in publishing speed?
>
> Thanks
>
> Libo
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB