We having issues that mirormaker not longer replicate newly created topics. It continues to replicate data for existing topics and but new topics doesn't get created on target cluster. ConsumerOffsetTracker shows that new topics are successfully consumed and Lag is 0. But those topics doesn't get created in target cluster. I also don't see mbeans for this new topic under kafka.producer.ProducerTopicMetrics.<topic name>metric. In logs I see warning for NotLeaderForPatition. but don't see major error. What else can we look to troubleshoot this further.
You said that ConsumerOffsetChecker shows that new topics are successfully consumed and the lag is 0. If that's the case, can you verify that there is data on the source cluster for these new topics? If there is no data at the source, MirrorMaker will only assign consumer streams to the new topic, but the lag will be 0.
yes, the data exists in source cluster, but not in target cluster. I can't replicate this problem in dev environment and it happens only in prod environment. I turned on debug logging, but not able to identify the problem. Basically, whenever I send data to new topic, I don't see any log messages from ProducerSendThread in mirrormaker log so they are not produced to target cluster. If I send more messages to same topic, the producer send thread kicks off and replicates the messages. But whatever messages send first time gets lost. How can I trouble shoot this problem further? Even this could be due to know issue https://issues.apache.org/jira/browse/KAFKA-1030, how can I confirm that? Is there config tweaking I can make to workaround this..? ConsumerOffsetChecks helps to track consumers. Its there any other tool we can use to track producers in mirrormaker. ?
Thanks in advance for help.
Thanks, Raja. On Fri, Sep 6, 2013 at 3:50 AM, Swapnil Ghike <[EMAIL PROTECTED]> wrote:
1) The consumer of mirror maker is successfully consuming all partitions of the newly created topic. 2) The producer of mirror maker is not producing the new messages immediately when the topic is created (observed from ProducerSendThread's log). 3) The producer of mirror maker will start producing the new messages when more messages are sent to the source cluster.
If 1) is true then KAFKA-1030 is excluded, since the consumer successfully recognize all the partitions and start consuming.
If both 2) and 3) is true, I would wonder if the batch size of the mirror maker producer is large and hence will not send until enough messages are accumulated at the producer queue.
Guozhang On Mon, Sep 9, 2013 at 2:36 PM, Rajasekar Elango <[EMAIL PROTECTED]>wrote:
1, 2, 3 all are true. We are using default value 200 for batch.num.messages and 5000ms queue.buffering.max.ms. I believe it should batch either if batch.num.messages is reached or queue.buffering.max.ms is reached.
I see log message "5000ms elapsed , Queue time reached. Sending. " on regular interval. But when I send messages to new topic, I always see "Handling 0 events" and it doesn't produce to target cluster. But when I resend it second time, I see "Handling x events" and starts producing. Any clues on how to debug further?
Raja. On Mon, Sep 9, 2013 at 6:02 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote: Thanks, Raja.
When you say "I send messages to new topic" I guess you mean that you send messages to the source cluster right? It may be due to the fact that producers of mirror make have not catched up with the mirror maker consumer.
When you say "I always see Handling 0 events" do you mean that you see this for both messages for the new topic and for the old topics, or it only shows this log for new topic?
Guozhang On Tue, Sep 10, 2013 at 7:47 AM, Rajasekar Elango <[EMAIL PROTECTED]>wrote:
1) When I say "I send messages to new topic" -> yes I am sending new messages to source cluster via console producer. 2) The log message "Handling 0 events" doesn't output topic name. But I would believe its for both old and new topics, because no other app is sending messages to source cluster other than me trying to test using console producer.
Thanks, Raja. On Tue, Sep 10, 2013 at 1:03 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote: Thanks, Raja.