Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - auto-delete empty, old topics?


+
Jason Rosenberg 2013-03-06, 23:15
+
Neha Narkhede 2013-03-06, 23:39
+
Jason Rosenberg 2013-03-07, 00:21
+
Neha Narkhede 2013-03-07, 02:08
+
Jason Rosenberg 2013-03-08, 05:13
+
Neha Narkhede 2013-03-08, 06:20
+
Jason Rosenberg 2013-03-14, 21:04
+
Jason Rosenberg 2013-03-14, 21:17
+
Neha Narkhede 2013-03-14, 21:39
+
Jason Rosenberg 2013-03-14, 22:20
Copy link to this message
-
Re: auto-delete empty, old topics?
Neha Narkhede 2013-03-14, 22:23
This could be a bug with the topic discovery logic in the wildcard
consumer. Please can you file a bug and attach your consumer logs there ?

Thanks,
Neha
On Thu, Mar 14, 2013 at 3:20 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Yes, you're description matches what I did.  And the brokers have been
> bounced many times since then (they are auto-deployed many times a day,
> etc.).  And the consumers also have been restarted many times since then.
>
> Could it be related to using the white-list topic selector, etc.?
>
> Let me know if you need me to look a the state of my zookeeper data....
>
> Jason
>
> On Thu, Mar 14, 2013 at 2:39 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > Jason,
> >
> > Let me see if I understood what you did here. In Kafka 0.7.2, you deleted
> > the Kafka log files from the server and bounced the broker. This
> should've
> > ideally deleted those topics from zookeeper, the consumer reads the same
> > zookeeper paths that the broker writes. Doing this should cause
> rebalancing
> > on the consumer and it should readjust its consumption to include the
> > remaining topics. But looks like that didn't happen somehow ?
> >
> > I agree that log line is redundant, I think it is removed in 0.8
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Mar 14, 2013 at 2:17 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Also,
> > >
> > > I see a bazillion consecutive log lines like this:
> > >
> > > 2013-03-14 19:54:13,306  INFO [Thread-4] consumer.ConsumerIterator -
> > > Clearing the current data chunk for this consumer iterator
> > >
> > > With the same content (not sure how useful that is!).
> > >
> > > Jason
> > >
> > > On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Hi Neha,
> > > >
> > > > So I did this, but I still see the full list of topics (most of which
> > > have
> > > > been deleted), in the consumer logs, e.g.:
> > > >
> > > > consumer.ZookeeperConsumerConnector -
> > > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb
> Topics
> > to
> > > > consume = List(<a bazillion topics, most of which have been deleted
> > > listed
> > > > here>)
> > > >
> > > > I select topics using the white list topic selector (using a
> wild-card
> > > > pattern, etc.).
> > > >
> > > > Thoughts?
> > > >
> > > > Does zookeeper keep separate info for the consumer, that needs to be
> > > > removed?
> > > >
> > > > Jason
> > > >
> > > > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > >wrote:
> > > >
> > > >> In Kafka 0.7.2, data log directory delete and then rolling restart
> > will
> > > >> work to get rid of the topic completely from zookeeper as well as
> the
> > > >> broker.
> > > >> This is because the broker registers the topics and partitions from
> > its
> > > >> data directory into zookeeper on startup.
> > > >>
> > > >> Thanks,
> > > >> Neha
> > > >>
> > > >>
> > > >> On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >>
> > > >> > Thanks Neha,
> > > >> >
> > > >> > So are you saying that on 0.7.2, to delete a topic I need only
> > remove
> > > >> it's
> > > >> > data log directory from each broker, and the restart the brokers?
> >  Is
> > > >> it ok
> > > >> > if it's a rolling restart?
> > > >> >
> > > >> > For some reason I thought I also had to do something on zookeeper
> > > >> servers
> > > >> > too, to remove the topics.  Is that not true?
> > > >> >
> > > >> > Jason
> > > >> >
> > > >> > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <
> > > [EMAIL PROTECTED]
> > > >> > >wrote:
> > > >> >
> > > >> > > >> The topics get automagically created if they don't exist.
>  Will
> > > >> > > that still exist in 0.8?
> > > >> > >
> > > >> > > In 0.8, this is controlled by the auto.create.topics.enable
> config
> > > on
> > > >> the
> > > >> > > brokers. If this is set to true, topics will be created when a
> > topic
> > > >> > > metadata request is sent for a new topic. This feature is

 
+
Jun Rao 2013-03-15, 15:18
+
Jason Rosenberg 2013-03-15, 21:19
+
Jun Rao 2013-03-16, 04:48
+
Jason Rosenberg 2013-03-17, 06:08
+
Jun Rao 2013-03-17, 16:09
+
Jason Rosenberg 2013-03-18, 06:11
+
Jun Rao 2013-03-18, 16:31
+
Jason Rosenberg 2013-03-18, 20:38
+
Neha Narkhede 2013-03-18, 21:29
+
Jason Rosenberg 2013-03-19, 01:12
+
Neha Narkhede 2013-03-18, 18:05
+
Jason Rosenberg 2013-03-18, 20:35
+
Jason Rosenberg 2013-03-08, 20:04