Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> auto-delete empty, old topics?

Copy link to this message
Re: auto-delete empty, old topics?
We plan to include support for deleting topics before the official release.
The JIRA tracking this feature is

This is mainly an admin tool and is proposed to be run manually. I think
your suggestion is to set some sort of expiration time on topics so when
all data gets garbage collected and the expiration time lapses, the topic
is automatically deleted.

For this expiration to work, we need some mechanism of knowing the size of
the data logs for that topic on all brokers. And only when its empty for
all partitions of that topic, we delete it. This requires some API that
will expose the data size of a topic on a broker, which we don't have right
now. It might be worth thinking about it. Do you mind posting your
suggestions for this tool on that JIRA ?

On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> I suspect this is not currently supported, but it seems to be for us a real
> use case.
> If we have a topic that is no longer receiving messages, and all messages
> have been removed from the brokers, after the log_retention_hours has
> expired, I'd love to have it then automatically remove the topic from
> kafka.
> If subsequently a new message arrives for that topic, it could just
> recreate the topic as originally, and continue on.
> Is this easy to implement?
> Has it been requested?
> Does it make sense?
> Should I file a jira?
> Currently, we had a test client that inadvertently created 1000's of
> topics.  The job of deleting all these appears to not be a cumbersome
> manual process (are there any quick links to the recommended way for doing
> this?).
> I am using 0.7.2, but will eventually upgrade to 0.8, once it's released to
> beta....
> Thanks,
> Jason