Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - auto-delete empty, old topics?


Copy link to this message
-
Re: auto-delete empty, old topics?
Neha Narkhede 2013-03-08, 06:20
In Kafka 0.7.2, data log directory delete and then rolling restart will
work to get rid of the topic completely from zookeeper as well as the
broker.
This is because the broker registers the topics and partitions from its
data directory into zookeeper on startup.

Thanks,
Neha
On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Thanks Neha,
>
> So are you saying that on 0.7.2, to delete a topic I need only remove it's
> data log directory from each broker, and the restart the brokers?  Is it ok
> if it's a rolling restart?
>
> For some reason I thought I also had to do something on zookeeper servers
> too, to remove the topics.  Is that not true?
>
> Jason
>
> On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > >> The topics get automagically created if they don't exist.  Will
> > that still exist in 0.8?
> >
> > In 0.8, this is controlled by the auto.create.topics.enable config on the
> > brokers. If this is set to true, topics will be created when a topic
> > metadata request is sent for a new topic. This feature is provided to aid
> > with the 0.7->0.8 migration tool and the mirror maker tools. These tools
> > copy data for almost all topics from one cluster to the other, so auto
> > creating topics makes it much easier to migrate data between clusters.
> >
> > >> What's the procedure for deleting topics in kafka 0.7.2?
> >
> > There is no official way to delete a topic. You have to delete the data
> > directory and bounce the brokers to take note of that.
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Thanks Neha,
> > >
> > > I added a comment to that ticket.
> > >
> > > Another question, I'm not sure if the behavior is different in 0.8 from
> > > 0.7.  But currently, I'm not predeclaring topics, before sending
> messages
> > > to topics.  The topics get automagically created if they don't exist.
> >  Will
> > > that still exist in 0.8?  If so, it should also be automatic (using a
> > > default expiration for all such topics), that they automagically go
> away
> > > after no longer being in use.
> > >
> > > What's the procedure for deleting topics in kafka 0.7.2?
> > >
> > > Jason
> > >
> > > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > We plan to include support for deleting topics before the official
> > > release.
> > > > The JIRA tracking this feature is
> > > > https://issues.apache.org/jira/browse/KAFKA-330
> > > >
> > > > This is mainly an admin tool and is proposed to be run manually. I
> > think
> > > > your suggestion is to set some sort of expiration time on topics so
> > when
> > > > all data gets garbage collected and the expiration time lapses, the
> > topic
> > > > is automatically deleted.
> > > >
> > > > For this expiration to work, we need some mechanism of knowing the
> size
> > > of
> > > > the data logs for that topic on all brokers. And only when its empty
> > for
> > > > all partitions of that topic, we delete it. This requires some API
> that
> > > > will expose the data size of a topic on a broker, which we don't have
> > > right
> > > > now. It might be worth thinking about it. Do you mind posting your
> > > > suggestions for this tool on that JIRA ?
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > I suspect this is not currently supported, but it seems to be for
> us
> > a
> > > > real
> > > > > use case.
> > > > >
> > > > > If we have a topic that is no longer receiving messages, and all
> > > messages
> > > > > have been removed from the brokers, after the log_retention_hours
> has
> > > > > expired, I'd love to have it then automatically remove the topic
> from
> > > > > kafka.
> > > > >
> > > > > If subsequently a new message arrives for that topic, it could just
> > > > > recreate the topic as originally, and continue on.