Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Topic creation on restart


Copy link to this message
-
Re: Topic creation on restart
Priya Matpadi 2013-11-05, 06:21
Can this unintentional topic creation be avoided by setting
auto.create.topics.enable=false?
On Mon, Nov 4, 2013 at 9:40 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Ok, so this can happen, even if the node has not been placed back into
> rotation, at the metadata vip?
>
>
> On Tue, Nov 5, 2013 at 12:11 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > It is probably due to the same metadata propagation issue.
> > https://issues.apache.org/jira/browse/KAFKA-1111 should fix that.
> >
> > Thanks,
> > Neha
> >
> >
> > On Mon, Nov 4, 2013 at 11:47 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Ok,
> > >
> > > After adding a delay before enabling a freshly started broker in the
> > > metadata vip that clients use, it seems to have drastically reduced the
> > > number of these topic creation requests.  However, not all of them.
> > >
> > > I still see (only some times) a handful of "Topic creation" log
> messages,
> > > that happen on a freshly started broker, during the time kafka has
> > started,
> > > but before the broker has been enabled in the metadata VIP.  Is there
> an
> > > explanation for this?
> > >
> > > Thanks!
> > >
> > > Jason
> > >
> > >
> > > On Fri, Nov 1, 2013 at 4:45 PM, Neha Narkhede <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > The mbeans are explained here -
> > > > http://kafka.apache.org/documentation.html#monitoring. Look for
> > > > *QueueTimeMs
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Fri, Nov 1, 2013 at 12:14 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Neha,
> > > > >
> > > > > This cluster has on the order of 750 topics.
> > > > >
> > > > > It looks like if I add a 20 second delay before placing a broker
> into
> > > the
> > > > > vip for metadata requests, it never seems to have this issue.  So
> I'm
> > > not
> > > > > sure about the 104 seconds number, other than that was how long the
> > > flood
> > > > > of "Topic creation" log messages went on for (over 500 of these).
> > > > >
> > > > > Which metric should I look at for 'high queue time'?
> > > > >
> > > > > Jason
> > > > >
> > > > >
> > > > > On Fri, Nov 1, 2013 at 5:54 AM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > It is proportional to the number of topics but still seems too
> > long.
> > > > Did
> > > > > > the broker have high queue time on all requests? Also how many
> > topics
> > > > > > existed on this cluster?
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > > On Oct 31, 2013 10:56 PM, "Jason Rosenberg" <[EMAIL PROTECTED]>
> > > wrote:
> > > > > >
> > > > > > > In this case, it appears to have gone on for 104 seconds.
>  Should
> > > it
> > > > > take
> > > > > > > that long?  It doesn't seem to always take this long.  I guess
> > one
> > > of
> > > > > the
> > > > > > > drawbacks of having a large number of topics (brings out the
> edge
> > > > > cases).
> > > > > > >
> > > > > > > Filed: https://issues.apache.org/jira/browse/KAFKA-1111
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Nov 1, 2013 at 1:17 AM, Neha Narkhede <
> > > > [EMAIL PROTECTED]
> > > > > > > >wrote:
> > > > > > >
> > > > > > > > >> Would it be possible for the broker to make sure all
> > metadata
> > > is
> > > > > > > > propagated, before completing the startup sequence, and
> taking
> > on
> > > > > > > metadata
> > > > > > > > requests?
> > > > > > > >
> > > > > > > > Yes, that is the flaw I mentioned before. We need to fix it.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Oct 31, 2013 at 9:56 PM, Jason Rosenberg <
> > > [EMAIL PROTECTED]
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Ok,
> > > > > > > > >
> > > > > > > > > So, I can safely ignore these, it sounds like.  I don't see
> > any
> > > > > > > > > corresponding logging around it subsequently not succeeding
> > to
> > > > > > actually
> > > > > > > > > create the topic in zk.
> > > > > > > > >
> > > > > > > > > Would it be possible for the broker to make sure all