Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Topic creation on restart


Copy link to this message
-
Re: Topic creation on restart
Jason Rosenberg 2013-11-01, 05:56
In this case, it appears to have gone on for 104 seconds.  Should it take
that long?  It doesn't seem to always take this long.  I guess one of the
drawbacks of having a large number of topics (brings out the edge cases).

Filed: https://issues.apache.org/jira/browse/KAFKA-1111
On Fri, Nov 1, 2013 at 1:17 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> >> Would it be possible for the broker to make sure all metadata is
> propagated, before completing the startup sequence, and taking on metadata
> requests?
>
> Yes, that is the flaw I mentioned before. We need to fix it.
>
>
> On Thu, Oct 31, 2013 at 9:56 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Ok,
> >
> > So, I can safely ignore these, it sounds like.  I don't see any
> > corresponding logging around it subsequently not succeeding to actually
> > create the topic in zk.
> >
> > Would it be possible for the broker to make sure all metadata is
> > propagated, before completing the startup sequence, and taking on
> metadata
> > requests?
> >
> > If I were to not place the newly started broker back into the vip for a
> > delay period after the kafka broker has started, would that solve this
> > issue?
> >
> > Jason
> >
> >
> > On Fri, Nov 1, 2013 at 12:39 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > This is probably because some metadata requests were issued to a newly
> > > started broker before the controller has propagated the metadata. Since
> > the
> > > broker doesn't have the metadata in the cache, it thinks the topic
> > doesn't
> > > exist and tries to create the topic. The creation will fail though
> since
> > > it's already registered in ZK. These INFO logging should be transient
> > > though.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Oct 31, 2013 at 8:41 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Some times during a rolling restart, I see lots of these messages
> > during
> > > > the restart (these happened in the logs of the 2nd server in the
> > cluster
> > > to
> > > > restart (broker 11), after having restarted started, and during the
> > time
> > > > the 3rd node (broker 9) is doing a controlled shutdown before
> > stopping).
> > > >
> > > > Is it creating new topics (it shouldn't have been!).  Or is it moving
> > > > topics around, or what?
> > > >
> > > > 2013-11-01 03:25:55,661  INFO [kafka-request-handler-7]
> > > admin.AdminUtils$ -
> > > > Topic creation { "partitions":{ "0":[ 9, 11 ] }, "version":1 }
> > > > 2013-11-01 03:26:01,577  INFO [kafka-request-handler-0]
> > > admin.AdminUtils$ -
> > > > Topic creation { "partitions":{ "0":[ 9, 10 ] }, "version":1 }
> > > > 2013-11-01 03:26:07,767  INFO [kafka-request-handler-1]
> > > admin.AdminUtils$ -
> > > > Topic creation { "partitions":{ "0":[ 9, 11 ] }, "version":1 }
> > > > 2013-11-01 03:26:07,823  INFO [kafka-request-handler-1]
> > > admin.AdminUtils$ -
> > > > Topic creation { "partitions":{ "0":[ 10, 11 ] }, "version":1 }
> > > > 2013-11-01 03:26:11,183  INFO [kafka-request-handler-2]
> > > admin.AdminUtils$ -
> > > > Topic creation { "partitions":{ "0":[ 10, 11 ] }, "version":1 }
> > > > 20
> > > >
> > >
> >
>