Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> auto-delete empty, old topics?


+
Jason Rosenberg 2013-03-06, 23:15
+
Neha Narkhede 2013-03-06, 23:39
+
Jason Rosenberg 2013-03-07, 00:21
+
Neha Narkhede 2013-03-07, 02:08
+
Jason Rosenberg 2013-03-08, 05:13
+
Neha Narkhede 2013-03-08, 06:20
+
Jason Rosenberg 2013-03-14, 21:04
+
Jason Rosenberg 2013-03-14, 21:17
+
Neha Narkhede 2013-03-14, 21:39
+
Jason Rosenberg 2013-03-14, 22:20
+
Neha Narkhede 2013-03-14, 22:23
Copy link to this message
-
Re: auto-delete empty, old topics?
Could you check if the following path for a deleted topic exists in ZK? It
should have no children.

/brokers/topics/[topic]

If this is the case, try manually removing those paths from ZK (when the
brokers and the consumers are down).

Thanks,

Jun

On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Hi Neha,
>
> So I did this, but I still see the full list of topics (most of which have
> been deleted), in the consumer logs, e.g.:
>
> consumer.ZookeeperConsumerConnector -
> samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
> consume = List(<a bazillion topics, most of which have been deleted listed
> here>)
>
> I select topics using the white list topic selector (using a wild-card
> pattern, etc.).
>
> Thoughts?
>
> Does zookeeper keep separate info for the consumer, that needs to be
> removed?
>
> Jason
>
> On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > In Kafka 0.7.2, data log directory delete and then rolling restart will
> > work to get rid of the topic completely from zookeeper as well as the
> > broker.
> > This is because the broker registers the topics and partitions from its
> > data directory into zookeeper on startup.
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Thanks Neha,
> > >
> > > So are you saying that on 0.7.2, to delete a topic I need only remove
> > it's
> > > data log directory from each broker, and the restart the brokers?  Is
> it
> > ok
> > > if it's a rolling restart?
> > >
> > > For some reason I thought I also had to do something on zookeeper
> servers
> > > too, to remove the topics.  Is that not true?
> > >
> > > Jason
> > >
> > > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > >> The topics get automagically created if they don't exist.  Will
> > > > that still exist in 0.8?
> > > >
> > > > In 0.8, this is controlled by the auto.create.topics.enable config on
> > the
> > > > brokers. If this is set to true, topics will be created when a topic
> > > > metadata request is sent for a new topic. This feature is provided to
> > aid
> > > > with the 0.7->0.8 migration tool and the mirror maker tools. These
> > tools
> > > > copy data for almost all topics from one cluster to the other, so
> auto
> > > > creating topics makes it much easier to migrate data between
> clusters.
> > > >
> > > > >> What's the procedure for deleting topics in kafka 0.7.2?
> > > >
> > > > There is no official way to delete a topic. You have to delete the
> data
> > > > directory and bounce the brokers to take note of that.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Thanks Neha,
> > > > >
> > > > > I added a comment to that ticket.
> > > > >
> > > > > Another question, I'm not sure if the behavior is different in 0.8
> > from
> > > > > 0.7.  But currently, I'm not predeclaring topics, before sending
> > > messages
> > > > > to topics.  The topics get automagically created if they don't
> exist.
> > > >  Will
> > > > > that still exist in 0.8?  If so, it should also be automatic
> (using a
> > > > > default expiration for all such topics), that they automagically go
> > > away
> > > > > after no longer being in use.
> > > > >
> > > > > What's the procedure for deleting topics in kafka 0.7.2?
> > > > >
> > > > > Jason
> > > > >
> > > > > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > We plan to include support for deleting topics before the
> official
> > > > > release.
> > > > > > The JIRA tracking this feature is
> > > > > > https://issues.apache.org/jira/browse/KAFKA-330
> > > > > >
> > > > > > This is mainly an admin tool and is proposed to be run manually.
> I
> > > > think
> > > > > > your suggestion is to set some sort of expiration time on topics

 
+
Jason Rosenberg 2013-03-15, 21:19
+
Jun Rao 2013-03-16, 04:48
+
Jason Rosenberg 2013-03-17, 06:08
+
Jun Rao 2013-03-17, 16:09
+
Jason Rosenberg 2013-03-18, 06:11
+
Jun Rao 2013-03-18, 16:31
+
Jason Rosenberg 2013-03-18, 20:38
+
Neha Narkhede 2013-03-18, 21:29
+
Jason Rosenberg 2013-03-19, 01:12
+
Neha Narkhede 2013-03-18, 18:05
+
Jason Rosenberg 2013-03-18, 20:35
+
Jason Rosenberg 2013-03-08, 20:04