Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Topic creation on restart


Copy link to this message
-
Re: Topic creation on restart
In this case, it appears to have gone on for 104 seconds.  Should it take
that long?  It doesn't seem to always take this long.  I guess one of the
drawbacks of having a large number of topics (brings out the edge cases).

Filed: https://issues.apache.org/jira/browse/KAFKA-1111
On Fri, Nov 1, 2013 at 1:17 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> >> Would it be possible for the broker to make sure all metadata is
> propagated, before completing the startup sequence, and taking on metadata
> requests?
>
> Yes, that is the flaw I mentioned before. We need to fix it.
>
>
> On Thu, Oct 31, 2013 at 9:56 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Ok,
> >
> > So, I can safely ignore these, it sounds like.  I don't see any
> > corresponding logging around it subsequently not succeeding to actually
> > create the topic in zk.
> >
> > Would it be possible for the broker to make sure all metadata is
> > propagated, before completing the startup sequence, and taking on
> metadata
> > requests?
> >
> > If I were to not place the newly started broker back into the vip for a
> > delay period after the kafka broker has started, would that solve this
> > issue?
> >
> > Jason
> >
> >
> > On Fri, Nov 1, 2013 at 12:39 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > This is probably because some metadata requests were issued to a newly
> > > started broker before the controller has propagated the metadata. Since
> > the
> > > broker doesn't have the metadata in the cache, it thinks the topic
> > doesn't
> > > exist and tries to create the topic. The creation will fail though
> since
> > > it's already registered in ZK. These INFO logging should be transient
> > > though.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Oct 31, 2013 at 8:41 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Some times during a rolling restart, I see lots of these messages
> > during
> > > > the restart (these happened in the logs of the 2nd server in the
> > cluster
> > > to
> > > > restart (broker 11), after having restarted started, and during the
> > time
> > > > the 3rd node (broker 9) is doing a controlled shutdown before
> > stopping).
> > > >
> > > > Is it creating new topics (it shouldn't have been!).  Or is it moving
> > > > topics around, or what?
> > > >
> > > > 2013-11-01 03:25:55,661  INFO [kafka-request-handler-7]
> > > admin.AdminUtils$ -
> > > > Topic creation { "partitions":{ "0":[ 9, 11 ] }, "version":1 }
> > > > 2013-11-01 03:26:01,577  INFO [kafka-request-handler-0]
> > > admin.AdminUtils$ -
> > > > Topic creation { "partitions":{ "0":[ 9, 10 ] }, "version":1 }
> > > > 2013-11-01 03:26:07,767  INFO [kafka-request-handler-1]
> > > admin.AdminUtils$ -
> > > > Topic creation { "partitions":{ "0":[ 9, 11 ] }, "version":1 }
> > > > 2013-11-01 03:26:07,823  INFO [kafka-request-handler-1]
> > > admin.AdminUtils$ -
> > > > Topic creation { "partitions":{ "0":[ 10, 11 ] }, "version":1 }
> > > > 2013-11-01 03:26:11,183  INFO [kafka-request-handler-2]
> > > admin.AdminUtils$ -
> > > > Topic creation { "partitions":{ "0":[ 10, 11 ] }, "version":1 }
> > > > 20
> > > >
> > >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB