I recently had a problem where one out of two of my brokers would not reboot due to a hardware failure. The broker was down for almost a week before the required part came in and was fixed by our datacenter tech. During that time, the live broker was able to handle all messages for all topics and partitions (which is awesome!). The first broker is now back, and is trying to catch up with the messages that it missed for the during. The lower volume topics are all caught up, but I have one high volume topic (around 40K msgs/sec) that is taking much longer. I just took a few samples of Replica-MaxLag to see how long it would take to catch up. Currently, it is behind about 12.5 million messages and is catching up at a rate of about 1600 msgs/sec. At that rate, it’ll take around 9 days before the replica is caught up to the leader.
Is there any way to speed this up?
Or, alternatively, I don’t actually care about this topic’s history. It is a new topic, and I know that it doesn't yet have any consumers. I’d be fine with instructing both brokers to drop old logs and just start from the top of the log. I could do this by manually deleting the topic (kafka data files and in zookeeper), but to do so properly with 0.8.0 I think I’d have to shut down the whole cluster, correct? I’d rather not do this, as another topic does have a consumer and I don’t want to lose messages for it.
During the period your high-volume topic is under-replicated you can temporarily try one or both of the following: - Increasing num.replica.fetchers (defaults is one) - If you don't have too many topic-partitions you can also increase replica.fetch.max.bytes. Right - or you could do a rolling bounce and change the retention settings (http://kafka.apache.org/documentation.html#brokerconfigs) of that topic to something low so it gets expired and then do another rolling bounce to remove the override.
Awesome! I just tried this one, bumped it up to 8 (12 cores on this broker box). It is now catching up at around 17K msgs/sec, which will mean it will finish in about 4 or 5 hours. I’ll check up on it again tomorrow.
That should do it, Thanks!
On Feb 5, 2014, at 5:04 PM, Joel Koshy <[EMAIL PROTECTED]> wrote:
Yes I think so - max fetch size defaults to 1 MB and num-replica fetchers to one which should be sufficient for most people. Setting those higher would typically lead to higher memory usage when there are a large number of topics and with num-fetchers it would multiply the number of socket connections between brokers (although that shouldn't be a big deal).
On Wed, Feb 05, 2014 at 03:42:23PM -0800, Jay Kreps wrote: