Assume A and B are two brokers in a kafka server. And there is long network Latency between A and B. For a partition with two replications, one replication Is assigned to A and the other is assigned to B. Number of acknowledge is set to one. Assume the partition is handled by broker A.
After a message is accepted and saved by A, ack will be sent and the message will be sent to broker B at the same time. But due to long network latency between A and B, it is possible that B lags behind A. In this case, is there a buffer on broker A that holds all messages to be written to B? Is there any limit on the lag between A and B?
Compare with the case when there is no network latency between A and B, any difference in publishing speed?
I'm assuming this is somewhat related to your previous question on cross-DC replication. This is not an ideal set up as mentioned there. If the replica lags then it will fall out of the "in-sync-replica" set. You could tune parameters that effectively allow a high (but bounded over time) lag between the replica and the leader. There is no buffer per se - the replicas just keep fetching from the leader (which would have appended the messages to its log).
If your producer has acks set to -1 (which means wait until all replicas in ISR have received the message) it would slow down your producer considerably. (You could use acks = 1 in that case though but with weaker guarantees - i.e., your message could be lost in failure scenarios).
Joel On Fri, Jun 28, 2013 at 5:41 PM, Yu, Libo <[EMAIL PROTECTED]> wrote:
There is "replica.lag.max.messages" (defaults to 4k) and "replica.lag.time.max.ms" (defaults to 10s). If a replica is behind the leader by either that many messages or hasn't sent any fetch requests within the lag config, then it falls out of ISR.
However, as mentioned earlier it is best to avoid a cross-DC replication set up. You should set the above configs to be reasonable values (the defaults should be reasonable, but it depends on your volume) and if you see replicas going out of ISR then address the underlying issue(s) - e.g., depending on what the issue is it could be enough to increase the number of replica fetcher threads. Increasing the replica message and time lag may help reduce ISR churns but it does not address the issue (which is replicas being unable to keep up). On Sat, Jun 29, 2013 at 10:47 AM, Yu, Libo <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext