Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> auto-delete empty, old topics?


+
Jason Rosenberg 2013-03-06, 23:15
+
Neha Narkhede 2013-03-06, 23:39
+
Jason Rosenberg 2013-03-07, 00:21
+
Neha Narkhede 2013-03-07, 02:08
+
Jason Rosenberg 2013-03-08, 05:13
+
Neha Narkhede 2013-03-08, 06:20
+
Jason Rosenberg 2013-03-14, 21:04
+
Jason Rosenberg 2013-03-14, 21:17
+
Neha Narkhede 2013-03-14, 21:39
+
Jason Rosenberg 2013-03-14, 22:20
+
Neha Narkhede 2013-03-14, 22:23
Copy link to this message
-
Re: auto-delete empty, old topics?
Could you check if the following path for a deleted topic exists in ZK? It
should have no children.

/brokers/topics/[topic]

If this is the case, try manually removing those paths from ZK (when the
brokers and the consumers are down).

Thanks,

Jun

On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Hi Neha,
>
> So I did this, but I still see the full list of topics (most of which have
> been deleted), in the consumer logs, e.g.:
>
> consumer.ZookeeperConsumerConnector -
> samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
> consume = List(<a bazillion topics, most of which have been deleted listed
> here>)
>
> I select topics using the white list topic selector (using a wild-card
> pattern, etc.).
>
> Thoughts?
>
> Does zookeeper keep separate info for the consumer, that needs to be
> removed?
>
> Jason
>
> On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > In Kafka 0.7.2, data log directory delete and then rolling restart will
> > work to get rid of the topic completely from zookeeper as well as the
> > broker.
> > This is because the broker registers the topics and partitions from its
> > data directory into zookeeper on startup.
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Thanks Neha,
> > >
> > > So are you saying that on 0.7.2, to delete a topic I need only remove
> > it's
> > > data log directory from each broker, and the restart the brokers?  Is
> it
> > ok
> > > if it's a rolling restart?
> > >
> > > For some reason I thought I also had to do something on zookeeper
> servers
> > > too, to remove the topics.  Is that not true?
> > >
> > > Jason
> > >
> > > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > >> The topics get automagically created if they don't exist.  Will
> > > > that still exist in 0.8?
> > > >
> > > > In 0.8, this is controlled by the auto.create.topics.enable config on
> > the
> > > > brokers. If this is set to true, topics will be created when a topic
> > > > metadata request is sent for a new topic. This feature is provided to
> > aid
> > > > with the 0.7->0.8 migration tool and the mirror maker tools. These
> > tools
> > > > copy data for almost all topics from one cluster to the other, so
> auto
> > > > creating topics makes it much easier to migrate data between
> clusters.
> > > >
> > > > >> What's the procedure for deleting topics in kafka 0.7.2?
> > > >
> > > > There is no official way to delete a topic. You have to delete the
> data
> > > > directory and bounce the brokers to take note of that.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Thanks Neha,
> > > > >
> > > > > I added a comment to that ticket.
> > > > >
> > > > > Another question, I'm not sure if the behavior is different in 0.8
> > from
> > > > > 0.7.  But currently, I'm not predeclaring topics, before sending
> > > messages
> > > > > to topics.  The topics get automagically created if they don't
> exist.
> > > >  Will
> > > > > that still exist in 0.8?  If so, it should also be automatic
> (using a
> > > > > default expiration for all such topics), that they automagically go
> > > away
> > > > > after no longer being in use.
> > > > >
> > > > > What's the procedure for deleting topics in kafka 0.7.2?
> > > > >
> > > > > Jason
> > > > >
> > > > > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > We plan to include support for deleting topics before the
> official
> > > > > release.
> > > > > > The JIRA tracking this feature is
> > > > > > https://issues.apache.org/jira/browse/KAFKA-330
> > > > > >
> > > > > > This is mainly an admin tool and is proposed to be run manually.
> I
> > > > think
> > > > > > your suggestion is to set some sort of expiration time on topics

 
+
Jason Rosenberg 2013-03-15, 21:19
+
Jun Rao 2013-03-16, 04:48
+
Jason Rosenberg 2013-03-17, 06:08
+
Jun Rao 2013-03-17, 16:09
+
Jason Rosenberg 2013-03-18, 06:11
+
Jun Rao 2013-03-18, 16:31
+
Jason Rosenberg 2013-03-18, 20:38
+
Neha Narkhede 2013-03-18, 21:29
+
Jason Rosenberg 2013-03-19, 01:12
+
Neha Narkhede 2013-03-18, 18:05
+
Jason Rosenberg 2013-03-18, 20:35
+
Jason Rosenberg 2013-03-08, 20:04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB