Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Topic creation on restart


Copy link to this message
-
Re: Topic creation on restart
Neha Narkhede 2013-11-01, 20:46
The mbeans are explained here -
http://kafka.apache.org/documentation.html#monitoring. Look for *QueueTimeMs

Thanks,
Neha
On Fri, Nov 1, 2013 at 12:14 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Neha,
>
> This cluster has on the order of 750 topics.
>
> It looks like if I add a 20 second delay before placing a broker into the
> vip for metadata requests, it never seems to have this issue.  So I'm not
> sure about the 104 seconds number, other than that was how long the flood
> of "Topic creation" log messages went on for (over 500 of these).
>
> Which metric should I look at for 'high queue time'?
>
> Jason
>
>
> On Fri, Nov 1, 2013 at 5:54 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > It is proportional to the number of topics but still seems too long. Did
> > the broker have high queue time on all requests? Also how many topics
> > existed on this cluster?
> >
> > Thanks,
> > Neha
> > On Oct 31, 2013 10:56 PM, "Jason Rosenberg" <[EMAIL PROTECTED]> wrote:
> >
> > > In this case, it appears to have gone on for 104 seconds.  Should it
> take
> > > that long?  It doesn't seem to always take this long.  I guess one of
> the
> > > drawbacks of having a large number of topics (brings out the edge
> cases).
> > >
> > > Filed: https://issues.apache.org/jira/browse/KAFKA-1111
> > >
> > >
> > > On Fri, Nov 1, 2013 at 1:17 AM, Neha Narkhede <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > >> Would it be possible for the broker to make sure all metadata is
> > > > propagated, before completing the startup sequence, and taking on
> > > metadata
> > > > requests?
> > > >
> > > > Yes, that is the flaw I mentioned before. We need to fix it.
> > > >
> > > >
> > > > On Thu, Oct 31, 2013 at 9:56 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Ok,
> > > > >
> > > > > So, I can safely ignore these, it sounds like.  I don't see any
> > > > > corresponding logging around it subsequently not succeeding to
> > actually
> > > > > create the topic in zk.
> > > > >
> > > > > Would it be possible for the broker to make sure all metadata is
> > > > > propagated, before completing the startup sequence, and taking on
> > > > metadata
> > > > > requests?
> > > > >
> > > > > If I were to not place the newly started broker back into the vip
> > for a
> > > > > delay period after the kafka broker has started, would that solve
> > this
> > > > > issue?
> > > > >
> > > > > Jason
> > > > >
> > > > >
> > > > > On Fri, Nov 1, 2013 at 12:39 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > This is probably because some metadata requests were issued to a
> > > newly
> > > > > > started broker before the controller has propagated the metadata.
> > > Since
> > > > > the
> > > > > > broker doesn't have the metadata in the cache, it thinks the
> topic
> > > > > doesn't
> > > > > > exist and tries to create the topic. The creation will fail
> though
> > > > since
> > > > > > it's already registered in ZK. These INFO logging should be
> > transient
> > > > > > though.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 31, 2013 at 8:41 PM, Jason Rosenberg <
> [EMAIL PROTECTED]
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Some times during a rolling restart, I see lots of these
> messages
> > > > > during
> > > > > > > the restart (these happened in the logs of the 2nd server in
> the
> > > > > cluster
> > > > > > to
> > > > > > > restart (broker 11), after having restarted started, and during
> > the
> > > > > time
> > > > > > > the 3rd node (broker 9) is doing a controlled shutdown before
> > > > > stopping).
> > > > > > >
> > > > > > > Is it creating new topics (it shouldn't have been!).  Or is it
> > > moving
> > > > > > > topics around, or what?
> > > > > > >
> > > > > > > 2013-11-01 03:25:55,661  INFO [kafka-request-handler-7]
> > > > > > admin.AdminUtils$ -
> > > > > > > Topic creation { "partitions":{ "0":[ 9, 11 ] }, "version":1 }
> > > > > > > 2013-11-01 03:26:01,577  INFO [kafka-request-handler-0]