Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> auto-delete empty, old topics?


Copy link to this message
-
Re: auto-delete empty, old topics?
Jun,

So, I connected to zookeeper just using telnet, and using the 4 letter
commands.

If I do a dump:

I do not see anything but valid topics, and valid consumer/owners mappings.

If I check watches, I see all the 1000's of bogus topics, e.c.:

wchc:
....
/brokers/topics/<obsolete-topic>
....

or

wchp
....
/brokers/topics/<obsolete-topic>.
0x383d6fe1b83f0008
0x383d6fe1b83f0009
0x7c3d6f9020d00041
0x833d6fe2826e0000
0x383d6fe1b83f000a
....

The kafka and zookeeper servers, as well as the consumer apps, are rolled
several times a day, automatically.

I'm not sure how to remove the watches, is that what I need to do?

Since this is a highly available system, can I bring down each zk server 1
at a time, and clear the bogus topics (or watches)?  How do I do this?

Jason

On Fri, Mar 15, 2013 at 8:17 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Could you check if the following path for a deleted topic exists in ZK? It
> should have no children.
>
> /brokers/topics/[topic]
>
> If this is the case, try manually removing those paths from ZK (when the
> brokers and the consumers are down).
>
> Thanks,
>
> Jun
>
> On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Hi Neha,
> >
> > So I did this, but I still see the full list of topics (most of which
> have
> > been deleted), in the consumer logs, e.g.:
> >
> > consumer.ZookeeperConsumerConnector -
> > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
> > consume = List(<a bazillion topics, most of which have been deleted
> listed
> > here>)
> >
> > I select topics using the white list topic selector (using a wild-card
> > pattern, etc.).
> >
> > Thoughts?
> >
> > Does zookeeper keep separate info for the consumer, that needs to be
> > removed?
> >
> > Jason
> >
> > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > In Kafka 0.7.2, data log directory delete and then rolling restart will
> > > work to get rid of the topic completely from zookeeper as well as the
> > > broker.
> > > This is because the broker registers the topics and partitions from its
> > > data directory into zookeeper on startup.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Thanks Neha,
> > > >
> > > > So are you saying that on 0.7.2, to delete a topic I need only remove
> > > it's
> > > > data log directory from each broker, and the restart the brokers?  Is
> > it
> > > ok
> > > > if it's a rolling restart?
> > > >
> > > > For some reason I thought I also had to do something on zookeeper
> > servers
> > > > too, to remove the topics.  Is that not true?
> > > >
> > > > Jason
> > > >
> > > > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > >> The topics get automagically created if they don't exist.  Will
> > > > > that still exist in 0.8?
> > > > >
> > > > > In 0.8, this is controlled by the auto.create.topics.enable config
> on
> > > the
> > > > > brokers. If this is set to true, topics will be created when a
> topic
> > > > > metadata request is sent for a new topic. This feature is provided
> to
> > > aid
> > > > > with the 0.7->0.8 migration tool and the mirror maker tools. These
> > > tools
> > > > > copy data for almost all topics from one cluster to the other, so
> > auto
> > > > > creating topics makes it much easier to migrate data between
> > clusters.
> > > > >
> > > > > >> What's the procedure for deleting topics in kafka 0.7.2?
> > > > >
> > > > > There is no official way to delete a topic. You have to delete the
> > data
> > > > > directory and bounce the brokers to take note of that.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >
> > > > > > Thanks Neha,
> > > > > >
> > > > > > I added a comment to that ticket.
> > > > > >
> > > > > > Another question, I'm not sure if the behavior is different in