I suspect this is not currently supported, but it seems to be for us a real use case.
If we have a topic that is no longer receiving messages, and all messages have been removed from the brokers, after the log_retention_hours has expired, I'd love to have it then automatically remove the topic from kafka.
If subsequently a new message arrives for that topic, it could just recreate the topic as originally, and continue on.
Is this easy to implement? Has it been requested? Does it make sense? Should I file a jira?
Currently, we had a test client that inadvertently created 1000's of topics. The job of deleting all these appears to not be a cumbersome manual process (are there any quick links to the recommended way for doing this?).
I am using 0.7.2, but will eventually upgrade to 0.8, once it's released to beta....
This is mainly an admin tool and is proposed to be run manually. I think your suggestion is to set some sort of expiration time on topics so when all data gets garbage collected and the expiration time lapses, the topic is automatically deleted.
For this expiration to work, we need some mechanism of knowing the size of the data logs for that topic on all brokers. And only when its empty for all partitions of that topic, we delete it. This requires some API that will expose the data size of a topic on a broker, which we don't have right now. It might be worth thinking about it. Do you mind posting your suggestions for this tool on that JIRA ?
Thanks, Neha On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
Another question, I'm not sure if the behavior is different in 0.8 from 0.7. But currently, I'm not predeclaring topics, before sending messages to topics. The topics get automagically created if they don't exist. Will that still exist in 0.8? If so, it should also be automatic (using a default expiration for all such topics), that they automagically go away after no longer being in use.
What's the procedure for deleting topics in kafka 0.7.2?
On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
In 0.8, this is controlled by the auto.create.topics.enable config on the brokers. If this is set to true, topics will be created when a topic metadata request is sent for a new topic. This feature is provided to aid with the 0.7->0.8 migration tool and the mirror maker tools. These tools copy data for almost all topics from one cluster to the other, so auto creating topics makes it much easier to migrate data between clusters. There is no official way to delete a topic. You have to delete the data directory and bounce the brokers to take note of that.
Thanks, Neha On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
In Kafka 0.7.2, data log directory delete and then rolling restart will work to get rid of the topic completely from zookeeper as well as the broker. This is because the broker registers the topics and partitions from its data directory into zookeeper on startup.
Thanks, Neha On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
I just want to add, that we have a use case where it is nice to able to have auto-created topics, as part of the design in regular operation. Thus, I'd hope the 'auto.create.topics.enable' has a purpose beyond just migrating data, etc! It should be a first class use case, in my view.
Let me see if I understood what you did here. In Kafka 0.7.2, you deleted the Kafka log files from the server and bounced the broker. This should've ideally deleted those topics from zookeeper, the consumer reads the same zookeeper paths that the broker writes. Doing this should cause rebalancing on the consumer and it should readjust its consumption to include the remaining topics. But looks like that didn't happen somehow ?
I agree that log line is redundant, I think it is removed in 0.8
Thanks, Neha On Thu, Mar 14, 2013 at 2:17 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
Yes, you're description matches what I did. And the brokers have been bounced many times since then (they are auto-deployed many times a day, etc.). And the consumers also have been restarted many times since then.
Could it be related to using the white-list topic selector, etc.?
Let me know if you need me to look a the state of my zookeeper data....
On Thu, Mar 14, 2013 at 2:39 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
Use zkCli.sh to see if there is any child node in ZK under /brokers/topics/deletedtopic. You should see none since you already removed those topics from brokers' log. If so, you can manually remove those paths in ZK using zkCli.sh (stop the consumers first). After the consumers restart, you should not see the the deleted topics in the log of the consumers any more.
consumer.ZookeeperConsumerConnector - samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to consume = List(<a bazillion topics, most of which have been deleted listed here>)
On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
So, looks like in 0.7, the way to correctly delete topics from the brokers and zookeeper is as follows -
1. Shutdown producers, consumers and brokers. 2. Delete the topic logs from the brokers 3. Delete the /brokers/topics/<topic> nodes from zookeeper 4. Restart the brokers 5. Restart producers and consumers.
Thanks, Neha On Mon, Mar 18, 2013 at 9:31 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
So we'll leave it as this (I won't file a bug). It is of course less than ideal to require everything to be brought down when deleting topics! Since we consider the cluster to be a high-availability resource. I assume in 0.8, it won't be necessary to bring everything down to delete a topic?
On Mon, Mar 18, 2013 at 11:05 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
One other clue, is that although those topics still show up under /brokers/topics/<topicname>/ they contain empty sub-nodes.....So zk knows they don't exist on any broker. So maybe that's the issue. Brokers are successfully removing themselves as serving a topic, but the topic itself remains. The consumers then just see the topics (even with no brokers listed) and continue to watch the topic.
On Mon, Mar 18, 2013 at 1:37 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
Jason Rosenberg 2013-03-19, 01:12
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation projects and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext