Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> auto-delete empty, old topics?


+
Jason Rosenberg 2013-03-06, 23:15
+
Neha Narkhede 2013-03-06, 23:39
+
Jason Rosenberg 2013-03-07, 00:21
+
Neha Narkhede 2013-03-07, 02:08
+
Jason Rosenberg 2013-03-08, 05:13
+
Neha Narkhede 2013-03-08, 06:20
+
Jason Rosenberg 2013-03-14, 21:04
+
Jason Rosenberg 2013-03-14, 21:17
+
Neha Narkhede 2013-03-14, 21:39
+
Jason Rosenberg 2013-03-14, 22:20
+
Neha Narkhede 2013-03-14, 22:23
+
Jun Rao 2013-03-15, 15:18
+
Jason Rosenberg 2013-03-15, 21:19
+
Jun Rao 2013-03-16, 04:48
Copy link to this message
-
Re: auto-delete empty, old topics?
Hi Jun, yes, we're using zk-based producers.  Producers have all been
restarted (they get cycled many times a day as well).

The specific producers that generated all those bogus topics have
definitely been restarted, etc.

Using zkCli.sh, I do see all the deleted topics, e.g.:

[zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
[<1000's of deleted topics>,<also all the remaining valid topics>]

I see also all the deleted topics under /consumers/<appname>/owners/

Jason

On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Are you using ZK-based producer? If so, those watches could be left by the
> producers if they haven't been restarted. Could you also use zkCli.sh to
> see if deleted topics are there in ZK?
>
> Thanks,
>
> Jun
>
> On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Jun,
> >
> > So, I connected to zookeeper just using telnet, and using the 4 letter
> > commands.
> >
> > If I do a dump:
> >
> > I do not see anything but valid topics, and valid consumer/owners
> mappings.
> >
> > If I check watches, I see all the 1000's of bogus topics, e.c.:
> >
> > wchc:
> > ....
> > /brokers/topics/<obsolete-topic>
> > ....
> >
> > or
> >
> > wchp
> > ....
> > /brokers/topics/<obsolete-topic>.
> > 0x383d6fe1b83f0008
> > 0x383d6fe1b83f0009
> > 0x7c3d6f9020d00041
> > 0x833d6fe2826e0000
> > 0x383d6fe1b83f000a
> > ....
> >
> > The kafka and zookeeper servers, as well as the consumer apps, are rolled
> > several times a day, automatically.
> >
> > I'm not sure how to remove the watches, is that what I need to do?
> >
> > Since this is a highly available system, can I bring down each zk server
> 1
> > at a time, and clear the bogus topics (or watches)?  How do I do this?
> >
> > Jason
> >
> >
> >
> > On Fri, Mar 15, 2013 at 8:17 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Could you check if the following path for a deleted topic exists in ZK?
> > It
> > > should have no children.
> > >
> > > /brokers/topics/[topic]
> > >
> > > If this is the case, try manually removing those paths from ZK (when
> the
> > > brokers and the consumers are down).
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Hi Neha,
> > > >
> > > > So I did this, but I still see the full list of topics (most of which
> > > have
> > > > been deleted), in the consumer logs, e.g.:
> > > >
> > > > consumer.ZookeeperConsumerConnector -
> > > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb
> Topics
> > to
> > > > consume = List(<a bazillion topics, most of which have been deleted
> > > listed
> > > > here>)
> > > >
> > > > I select topics using the white list topic selector (using a
> wild-card
> > > > pattern, etc.).
> > > >
> > > > Thoughts?
> > > >
> > > > Does zookeeper keep separate info for the consumer, that needs to be
> > > > removed?
> > > >
> > > > Jason
> > > >
> > > > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > In Kafka 0.7.2, data log directory delete and then rolling restart
> > will
> > > > > work to get rid of the topic completely from zookeeper as well as
> the
> > > > > broker.
> > > > > This is because the broker registers the topics and partitions from
> > its
> > > > > data directory into zookeeper on startup.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >
> > > > > > Thanks Neha,
> > > > > >
> > > > > > So are you saying that on 0.7.2, to delete a topic I need only
> > remove
> > > > > it's
> > > > > > data log directory from each broker, and the restart the brokers?
> >  Is
> > > > it
> > > > > ok
> > > > > > if it's a rolling restart?
> > > > > >
> > > > > > For some reason I thought I also had to do something on zookeeper
> > > > servers
> > > > > > too, to remove the topics.  Is that not true?
> > >
 
+
Jun Rao 2013-03-17, 16:09
+
Jason Rosenberg 2013-03-18, 06:11
+
Jun Rao 2013-03-18, 16:31
+
Jason Rosenberg 2013-03-18, 20:38
+
Neha Narkhede 2013-03-18, 21:29
+
Jason Rosenberg 2013-03-19, 01:12
+
Neha Narkhede 2013-03-18, 18:05
+
Jason Rosenberg 2013-03-18, 20:35
+
Jason Rosenberg 2013-03-08, 20:04