Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> auto-delete empty, old topics?


Copy link to this message
-
Re: auto-delete empty, old topics?
Thanks Jun,

I have nothing listed under /brokers/topics/deletedtopic, etc....so that
doesn't appear to be the issue.

I will try removing now the unwanted topics under /brokers/topics/.

Jason
On Mon, Mar 18, 2013 at 9:31 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Jason,
>
> This is mainly a problem that we don't have a formal way of deleting a
> topic in 0.7, which we are trying to fix in 0.8.
>
> The extra watchers on those deleted topics are likely registered by the
> producers. They should be gone once /brokers/topics/deletedtopic are
> removed from ZK.
>
> You probably don't need to clean /consumers/<appname>/owners.
>
> Thanks,
>
> Jun
>
>
> On Sun, Mar 17, 2013 at 11:11 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
>
> > Jun,
> >
> > There are indeed no nodes under /brokers/topics/deletedtopic
> >
> > Also, do I need to remove the deleted apps from the
> > /consumers/<appname>/owners path?
> >
> > So, should this be considered a bug, or not?  Essentially, each broker
> > knows what topics it has, but the info seems to have not propagated to zk
> > at all?  Is that right?
> >
> > Although from the telnet interface, the removed topics don't show up
> under
> > a 'dump' command, but they do under wchc, etc.  What's that about?
> >
> > Should the original procedure have been to: (1) stop all consumers, then
> > (2) remove the topic files on the brokers, then (3) restart the brokers,
> > then (4) restart the consumers?
> >
> > The consumers have been up and down since the topic files were removed,
> but
> > just not ever at the same time the brokers were down....
> >
> > Jason
> >
> >
> > On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Jason,
> > >
> > > Use zkCli.sh to see if there is any child node in ZK under
> > > /brokers/topics/deletedtopic. You should see none since you already
> > removed
> > > those topics from brokers' log. If so, you can manually remove those
> > paths
> > > in ZK using zkCli.sh (stop the consumers first). After the consumers
> > > restart, you should not see the the deleted topics in the log of the
> > > consumers any more.
> > >
> > > consumer.ZookeeperConsumerConnector -
> > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics
> to
> > > consume = List(<a bazillion topics, most of which have been deleted
> > listed
> > > here>)
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Hi Jun, yes, we're using zk-based producers.  Producers have all been
> > > > restarted (they get cycled many times a day as well).
> > > >
> > > > The specific producers that generated all those bogus topics have
> > > > definitely been restarted, etc.
> > > >
> > > > Using zkCli.sh, I do see all the deleted topics, e.g.:
> > > >
> > > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
> > > > [<1000's of deleted topics>,<also all the remaining valid topics>]
> > > >
> > > > I see also all the deleted topics under /consumers/<appname>/owners/
> > > >
> > > > Jason
> > > >
> > > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Are you using ZK-based producer? If so, those watches could be left
> > by
> > > > the
> > > > > producers if they haven't been restarted. Could you also use
> zkCli.sh
> > > to
> > > > > see if deleted topics are there in ZK?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > So, I connected to zookeeper just using telnet, and using the 4
> > > letter
> > > > > > commands.
> > > > > >
> > > > > > If I do a dump:
> > > > > >
> > > > > > I do not see anything but valid topics, and valid consumer/owners
> > > > > mappings.
> > > > > >
> > > > > > If I check watches, I see all the 1000's of bogus topics, e.c.:
> > > > > >
> > > > > > wchc:
> > > > > > ....
> >
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB