Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Mirror maker doesn't replicate new topics


Copy link to this message
-
Re: Mirror maker doesn't replicate new topics
Thanks Guozhang, Yes we had set to 'largest' and changing it to 'smallest'
resolved the issue. So it was due to the jira
https://issues.apache.org/jira/browse/KAFKA-1006

Thanks,
Raja.
On Tue, Sep 10, 2013 at 1:18 PM, Guozhang Wang <[EMAIL PROTECTED]> wrote:

> Oh got it. Did you set auto.offset.reset = smallest or largest? If it is
> largest it could be due to this bug:
>
> https://issues.apache.org/jira/browse/KAFKA-1006
>
> Guozhang
>
>
>
> On Tue, Sep 10, 2013 at 10:09 AM, Rajasekar Elango
> <[EMAIL PROTECTED]>wrote:
>
> > Hi Guozhang ,
> >
> > 1) When I say "I send messages to new topic" -> yes I am sending new
> > messages to source cluster via console producer.
> > 2) The log message "Handling 0 events" doesn't output topic name. But I
> > would believe its for both old and new topics, because no other app is
> > sending messages to source cluster other than me trying to test using
> > console producer.
> >
> > Thanks,
> > Raja.
> >
> >
> > On Tue, Sep 10, 2013 at 1:03 PM, Guozhang Wang <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hi Raja,
> > >
> > > When you say "I send messages to new topic" I guess you mean that you
> > send
> > > messages to the source cluster right? It may be due to the fact that
> > > producers of mirror make have not catched up with the mirror maker
> > > consumer.
> > >
> > > When you say "I always see Handling 0 events" do you mean that you see
> > this
> > > for both messages for the new topic and for the old topics, or it only
> > > shows this log for new topic?
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Sep 10, 2013 at 7:47 AM, Rajasekar Elango <
> > [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Thanks Guozhang,
> > > >
> > > > 1, 2, 3 all are true. We are using default value 200 for
> > > batch.num.messages
> > > > and 5000ms queue.buffering.max.ms. I believe it should batch either
> if
> > > > batch.num.messages is reached or queue.buffering.max.ms is reached.
> > > >
> > > > I see log message "5000ms elapsed , Queue time reached. Sending.  "
>  on
> > > > regular interval. But when I send messages to new topic, I always see
> > > > "Handling 0 events" and it doesn't produce to target cluster. But
> when
> > I
> > > > resend it second time, I see "Handling x events" and starts
> producing.
> > > Any
> > > > clues on how to debug further?
> > > >
> > > > Thanks,
> > > >
> > > > Raja.
> > > >
> > > >
> > > > On Mon, Sep 9, 2013 at 6:02 PM, Guozhang Wang <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Hi Raja,
> > > > >
> > > > > So just to summarize the scenario:
> > > > >
> > > > > 1) The consumer of mirror maker is successfully consuming all
> > > partitions
> > > > of
> > > > > the newly created topic.
> > > > > 2) The producer of mirror maker is not producing the new messages
> > > > > immediately when the topic is created (observed from
> > > ProducerSendThread's
> > > > > log).
> > > > > 3) The producer of mirror maker will start producing the new
> messages
> > > > when
> > > > > more messages are sent to the source cluster.
> > > > >
> > > > > If 1) is true then KAFKA-1030 is excluded, since the consumer
> > > > successfully
> > > > > recognize all the partitions and start consuming.
> > > > >
> > > > > If both 2) and 3) is true, I would wonder if the batch size of the
> > > mirror
> > > > > maker producer is large and hence will not send until enough
> messages
> > > are
> > > > > accumulated at the producer queue.
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > > On Mon, Sep 9, 2013 at 2:36 PM, Rajasekar Elango <
> > > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > yes, the data exists in source cluster, but not in target
> cluster.
> > I
> > > > > can't
> > > > > > replicate this problem in dev environment and it happens only in
> > prod
> > > > > > environment. I turned on debug logging, but not able to identify
> >  the
> > > > > > problem. Basically, whenever I send data to new topic, I don't
> see
> > > any
> > > > > log
> > > > > > messages from ProducerSendThread in mirrormaker log so they are

Thanks,
Raja.