Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Topic creation on restart


Copy link to this message
-
Re: Topic creation on restart
Ok,

I updated https://issues.apache.org/jira/browse/KAFKA-1111 with this
scenario also.

Jason
On Tue, Nov 5, 2013 at 9:11 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> >. Could a fetch request from a consumer cause a Topic creation request
> (seems
> implausible).....
>
> Yes, that seems like a way the broker can get into this situation.
>
> Thanks,
> Neha
>
>
> On Tue, Nov 5, 2013 at 4:58 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > I don't  know if I have a way to see the access logs on the
> LB......(still
> > trying to track that down).
> >
> > One thing I do see though, is that there are fetch requests from
> consumers,
> > that are then followed by these Topic creation log messages, e.g.
> (replaced
> > some of the specific strings in this log line :))
> >
> > 2013-11-04 23:28:07,828  WARN [kafka-request-handler-1] server.KafkaApis
> -
> > [KafkaApi-10] Fetch request with correlation id 0 from client
> >
> >
> my-conumser-app-ConsumerFetcherThread-my-consumer-app_myhost.mydc.mycompany-1383602435614-2397d1ab-0-10
> > on partition [mytopic,0] failed due to Partition [mytopic,0] doesn't
> exist
> > on 10
> > 2013-11-04 23:28:07,847  INFO [kafka-request-handler-7]
> admin.AdminUtils$ -
> > Topic creation { "partitions":{ "0":[ 10, 9 ] }, "version":1 }
> >
> > It's hard to say whether there's a correlation between these 2 log lines
> > (since the 'Topic creation' log line doesn't include the name of the
> topic.
> >  Shouldn't it?).
> >
> > Could a fetch request from a consumer cause a Topic creation request
> (seems
> > implausible).....
> >
> > Jason
> >
> >
> > On Tue, Nov 5, 2013 at 9:51 AM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Ok, so this can happen, even if the node has not been placed back into
> > > rotation, at the metadata vip?
> > >
> > > Hmmm... if the broker is not placed in the metadata vip, how did it end
> > up
> > > receiving metadata requests? You may want to investigate that by
> checking
> > > the public access logs.
> > >
> > > Thanks,
> > > Neha
> > > On Nov 4, 2013 9:41 PM, "Jason Rosenberg" <[EMAIL PROTECTED]> wrote:
> > >
> > > > Ok, so this can happen, even if the node has not been placed back
> into
> > > > rotation, at the metadata vip?
> > > >
> > > >
> > > > On Tue, Nov 5, 2013 at 12:11 AM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > It is probably due to the same metadata propagation issue.
> > > > > https://issues.apache.org/jira/browse/KAFKA-1111 should fix that.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Mon, Nov 4, 2013 at 11:47 AM, Jason Rosenberg <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > > >
> > > > > > Ok,
> > > > > >
> > > > > > After adding a delay before enabling a freshly started broker in
> > the
> > > > > > metadata vip that clients use, it seems to have drastically
> reduced
> > > the
> > > > > > number of these topic creation requests.  However, not all of
> them.
> > > > > >
> > > > > > I still see (only some times) a handful of "Topic creation" log
> > > > messages,
> > > > > > that happen on a freshly started broker, during the time kafka
> has
> > > > > started,
> > > > > > but before the broker has been enabled in the metadata VIP.  Is
> > there
> > > > an
> > > > > > explanation for this?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Jason
> > > > > >
> > > > > >
> > > > > > On Fri, Nov 1, 2013 at 4:45 PM, Neha Narkhede <
> > > [EMAIL PROTECTED]
> > > > > > >wrote:
> > > > > >
> > > > > > > The mbeans are explained here -
> > > > > > > http://kafka.apache.org/documentation.html#monitoring. Look
> for
> > > > > > > *QueueTimeMs
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Neha
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Nov 1, 2013 at 12:14 PM, Jason Rosenberg <
> > [EMAIL PROTECTED]
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Neha,
> > > > > > > >
> > > > > > > > This cluster has on the order of 750 topics.
> > > > > > > >
> > >
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB