Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Topic creation on restart


+
Jason Rosenberg 2013-11-01, 03:41
+
Neha Narkhede 2013-11-01, 04:35
+
Jun Rao 2013-11-01, 04:40
+
Jason Rosenberg 2013-11-01, 04:56
+
Neha Narkhede 2013-11-01, 05:18
+
Jason Rosenberg 2013-11-01, 05:56
+
Neha Narkhede 2013-11-01, 09:54
+
Jason Rosenberg 2013-11-01, 19:14
+
Neha Narkhede 2013-11-01, 20:46
+
Jason Rosenberg 2013-11-04, 19:48
+
Neha Narkhede 2013-11-05, 05:12
+
Jason Rosenberg 2013-11-05, 05:41
+
Priya Matpadi 2013-11-05, 06:21
+
Neha Narkhede 2013-11-05, 14:54
+
Neha Narkhede 2013-11-05, 14:54
Copy link to this message
-
Re: Topic creation on restart
I don't  know if I have a way to see the access logs on the LB......(still
trying to track that down).

One thing I do see though, is that there are fetch requests from consumers,
that are then followed by these Topic creation log messages, e.g. (replaced
some of the specific strings in this log line :))

2013-11-04 23:28:07,828  WARN [kafka-request-handler-1] server.KafkaApis -
[KafkaApi-10] Fetch request with correlation id 0 from client
my-conumser-app-ConsumerFetcherThread-my-consumer-app_myhost.mydc.mycompany-1383602435614-2397d1ab-0-10
on partition [mytopic,0] failed due to Partition [mytopic,0] doesn't exist
on 10
2013-11-04 23:28:07,847  INFO [kafka-request-handler-7] admin.AdminUtils$ -
Topic creation { "partitions":{ "0":[ 10, 9 ] }, "version":1 }

It's hard to say whether there's a correlation between these 2 log lines
(since the 'Topic creation' log line doesn't include the name of the topic.
 Shouldn't it?).

Could a fetch request from a consumer cause a Topic creation request (seems
implausible).....

Jason
On Tue, Nov 5, 2013 at 9:51 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Ok, so this can happen, even if the node has not been placed back into
> rotation, at the metadata vip?
>
> Hmmm... if the broker is not placed in the metadata vip, how did it end up
> receiving metadata requests? You may want to investigate that by checking
> the public access logs.
>
> Thanks,
> Neha
> On Nov 4, 2013 9:41 PM, "Jason Rosenberg" <[EMAIL PROTECTED]> wrote:
>
> > Ok, so this can happen, even if the node has not been placed back into
> > rotation, at the metadata vip?
> >
> >
> > On Tue, Nov 5, 2013 at 12:11 AM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > It is probably due to the same metadata propagation issue.
> > > https://issues.apache.org/jira/browse/KAFKA-1111 should fix that.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Mon, Nov 4, 2013 at 11:47 AM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Ok,
> > > >
> > > > After adding a delay before enabling a freshly started broker in the
> > > > metadata vip that clients use, it seems to have drastically reduced
> the
> > > > number of these topic creation requests.  However, not all of them.
> > > >
> > > > I still see (only some times) a handful of "Topic creation" log
> > messages,
> > > > that happen on a freshly started broker, during the time kafka has
> > > started,
> > > > but before the broker has been enabled in the metadata VIP.  Is there
> > an
> > > > explanation for this?
> > > >
> > > > Thanks!
> > > >
> > > > Jason
> > > >
> > > >
> > > > On Fri, Nov 1, 2013 at 4:45 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > The mbeans are explained here -
> > > > > http://kafka.apache.org/documentation.html#monitoring. Look for
> > > > > *QueueTimeMs
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Fri, Nov 1, 2013 at 12:14 PM, Jason Rosenberg <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > > >
> > > > > > Neha,
> > > > > >
> > > > > > This cluster has on the order of 750 topics.
> > > > > >
> > > > > > It looks like if I add a 20 second delay before placing a broker
> > into
> > > > the
> > > > > > vip for metadata requests, it never seems to have this issue.  So
> > I'm
> > > > not
> > > > > > sure about the 104 seconds number, other than that was how long
> the
> > > > flood
> > > > > > of "Topic creation" log messages went on for (over 500 of these).
> > > > > >
> > > > > > Which metric should I look at for 'high queue time'?
> > > > > >
> > > > > > Jason
> > > > > >
> > > > > >
> > > > > > On Fri, Nov 1, 2013 at 5:54 AM, Neha Narkhede <
> > > [EMAIL PROTECTED]
> > > > > > >wrote:
> > > > > >
> > > > > > > It is proportional to the number of topics but still seems too
> > > long.
> > > > > Did
> > > > > > > the broker have high queue time on all requests? Also how many
> > > topics
> > > > > > > existed on this cluster?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Neha

 
+
Neha Narkhede 2013-11-06, 02:12
+
Jason Rosenberg 2013-11-06, 18:10
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB