Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> auto-delete empty, old topics?


+
Jason Rosenberg 2013-03-06, 23:15
+
Neha Narkhede 2013-03-06, 23:39
+
Jason Rosenberg 2013-03-07, 00:21
+
Neha Narkhede 2013-03-07, 02:08
Copy link to this message
-
Re: auto-delete empty, old topics?
Thanks Neha,

So are you saying that on 0.7.2, to delete a topic I need only remove it's
data log directory from each broker, and the restart the brokers?  Is it ok
if it's a rolling restart?

For some reason I thought I also had to do something on zookeeper servers
too, to remove the topics.  Is that not true?

Jason

On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> >> The topics get automagically created if they don't exist.  Will
> that still exist in 0.8?
>
> In 0.8, this is controlled by the auto.create.topics.enable config on the
> brokers. If this is set to true, topics will be created when a topic
> metadata request is sent for a new topic. This feature is provided to aid
> with the 0.7->0.8 migration tool and the mirror maker tools. These tools
> copy data for almost all topics from one cluster to the other, so auto
> creating topics makes it much easier to migrate data between clusters.
>
> >> What's the procedure for deleting topics in kafka 0.7.2?
>
> There is no official way to delete a topic. You have to delete the data
> directory and bounce the brokers to take note of that.
>
> Thanks,
> Neha
>
>
> On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Thanks Neha,
> >
> > I added a comment to that ticket.
> >
> > Another question, I'm not sure if the behavior is different in 0.8 from
> > 0.7.  But currently, I'm not predeclaring topics, before sending messages
> > to topics.  The topics get automagically created if they don't exist.
>  Will
> > that still exist in 0.8?  If so, it should also be automatic (using a
> > default expiration for all such topics), that they automagically go away
> > after no longer being in use.
> >
> > What's the procedure for deleting topics in kafka 0.7.2?
> >
> > Jason
> >
> > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > We plan to include support for deleting topics before the official
> > release.
> > > The JIRA tracking this feature is
> > > https://issues.apache.org/jira/browse/KAFKA-330
> > >
> > > This is mainly an admin tool and is proposed to be run manually. I
> think
> > > your suggestion is to set some sort of expiration time on topics so
> when
> > > all data gets garbage collected and the expiration time lapses, the
> topic
> > > is automatically deleted.
> > >
> > > For this expiration to work, we need some mechanism of knowing the size
> > of
> > > the data logs for that topic on all brokers. And only when its empty
> for
> > > all partitions of that topic, we delete it. This requires some API that
> > > will expose the data size of a topic on a broker, which we don't have
> > right
> > > now. It might be worth thinking about it. Do you mind posting your
> > > suggestions for this tool on that JIRA ?
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > I suspect this is not currently supported, but it seems to be for us
> a
> > > real
> > > > use case.
> > > >
> > > > If we have a topic that is no longer receiving messages, and all
> > messages
> > > > have been removed from the brokers, after the log_retention_hours has
> > > > expired, I'd love to have it then automatically remove the topic from
> > > > kafka.
> > > >
> > > > If subsequently a new message arrives for that topic, it could just
> > > > recreate the topic as originally, and continue on.
> > > >
> > > > Is this easy to implement?
> > > > Has it been requested?
> > > > Does it make sense?
> > > > Should I file a jira?
> > > >
> > > > Currently, we had a test client that inadvertently created 1000's of
> > > > topics.  The job of deleting all these appears to not be a cumbersome
> > > > manual process (are there any quick links to the recommended way for
> > > doing
> > > > this?).
> > > >
> > > > I am using 0.7.2, but will eventually upgrade to 0.8, once it's
> > released
> > > to
> > > > beta....
> > > >
> > >
 
+
Neha Narkhede 2013-03-08, 06:20
+
Jason Rosenberg 2013-03-14, 21:04
+
Jason Rosenberg 2013-03-14, 21:17
+
Neha Narkhede 2013-03-14, 21:39
+
Jason Rosenberg 2013-03-14, 22:20
+
Neha Narkhede 2013-03-14, 22:23
+
Jun Rao 2013-03-15, 15:18
+
Jason Rosenberg 2013-03-15, 21:19
+
Jun Rao 2013-03-16, 04:48
+
Jason Rosenberg 2013-03-17, 06:08
+
Jun Rao 2013-03-17, 16:09
+
Jason Rosenberg 2013-03-18, 06:11
+
Jun Rao 2013-03-18, 16:31
+
Jason Rosenberg 2013-03-18, 20:38
+
Neha Narkhede 2013-03-18, 21:29
+
Jason Rosenberg 2013-03-19, 01:12
+
Neha Narkhede 2013-03-18, 18:05
+
Jason Rosenberg 2013-03-18, 20:35
+
Jason Rosenberg 2013-03-08, 20:04