Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> auto-delete empty, old topics?


Copy link to this message
-
Re: auto-delete empty, old topics?
Jason,

This is mainly a problem that we don't have a formal way of deleting a
topic in 0.7, which we are trying to fix in 0.8.

The extra watchers on those deleted topics are likely registered by the
producers. They should be gone once /brokers/topics/deletedtopic are
removed from ZK.

You probably don't need to clean /consumers/<appname>/owners.

Thanks,

Jun
On Sun, Mar 17, 2013 at 11:11 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Jun,
>
> There are indeed no nodes under /brokers/topics/deletedtopic
>
> Also, do I need to remove the deleted apps from the
> /consumers/<appname>/owners path?
>
> So, should this be considered a bug, or not?  Essentially, each broker
> knows what topics it has, but the info seems to have not propagated to zk
> at all?  Is that right?
>
> Although from the telnet interface, the removed topics don't show up under
> a 'dump' command, but they do under wchc, etc.  What's that about?
>
> Should the original procedure have been to: (1) stop all consumers, then
> (2) remove the topic files on the brokers, then (3) restart the brokers,
> then (4) restart the consumers?
>
> The consumers have been up and down since the topic files were removed, but
> just not ever at the same time the brokers were down....
>
> Jason
>
>
> On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Jason,
> >
> > Use zkCli.sh to see if there is any child node in ZK under
> > /brokers/topics/deletedtopic. You should see none since you already
> removed
> > those topics from brokers' log. If so, you can manually remove those
> paths
> > in ZK using zkCli.sh (stop the consumers first). After the consumers
> > restart, you should not see the the deleted topics in the log of the
> > consumers any more.
> >
> > consumer.ZookeeperConsumerConnector -
> > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
> > consume = List(<a bazillion topics, most of which have been deleted
> listed
> > here>)
> >
> > Thanks,
> >
> > Jun
> >
> > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi Jun, yes, we're using zk-based producers.  Producers have all been
> > > restarted (they get cycled many times a day as well).
> > >
> > > The specific producers that generated all those bogus topics have
> > > definitely been restarted, etc.
> > >
> > > Using zkCli.sh, I do see all the deleted topics, e.g.:
> > >
> > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
> > > [<1000's of deleted topics>,<also all the remaining valid topics>]
> > >
> > > I see also all the deleted topics under /consumers/<appname>/owners/
> > >
> > > Jason
> > >
> > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > > > Are you using ZK-based producer? If so, those watches could be left
> by
> > > the
> > > > producers if they haven't been restarted. Could you also use zkCli.sh
> > to
> > > > see if deleted topics are there in ZK?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > So, I connected to zookeeper just using telnet, and using the 4
> > letter
> > > > > commands.
> > > > >
> > > > > If I do a dump:
> > > > >
> > > > > I do not see anything but valid topics, and valid consumer/owners
> > > > mappings.
> > > > >
> > > > > If I check watches, I see all the 1000's of bogus topics, e.c.:
> > > > >
> > > > > wchc:
> > > > > ....
> > > > > /brokers/topics/<obsolete-topic>
> > > > > ....
> > > > >
> > > > > or
> > > > >
> > > > > wchp
> > > > > ....
> > > > > /brokers/topics/<obsolete-topic>.
> > > > > 0x383d6fe1b83f0008
> > > > > 0x383d6fe1b83f0009
> > > > > 0x7c3d6f9020d00041
> > > > > 0x833d6fe2826e0000
> > > > > 0x383d6fe1b83f000a
> > > > > ....
> > > > >
> > > > > The kafka and zookeeper servers, as well as the consumer apps, are
> > > rolled
> > > > > several times a day, automatically.
> > > > >
> > > > > I'm not sure how to remove the watches, is that what I need to do?