Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> auto-delete empty, old topics?


+
Jason Rosenberg 2013-03-06, 23:15
+
Neha Narkhede 2013-03-06, 23:39
+
Jason Rosenberg 2013-03-07, 00:21
+
Neha Narkhede 2013-03-07, 02:08
+
Jason Rosenberg 2013-03-08, 05:13
+
Neha Narkhede 2013-03-08, 06:20
+
Jason Rosenberg 2013-03-14, 21:04
+
Jason Rosenberg 2013-03-14, 21:17
+
Neha Narkhede 2013-03-14, 21:39
+
Jason Rosenberg 2013-03-14, 22:20
+
Neha Narkhede 2013-03-14, 22:23
+
Jun Rao 2013-03-15, 15:18
+
Jason Rosenberg 2013-03-15, 21:19
+
Jun Rao 2013-03-16, 04:48
+
Jason Rosenberg 2013-03-17, 06:08
+
Jun Rao 2013-03-17, 16:09
+
Jason Rosenberg 2013-03-18, 06:11
+
Jun Rao 2013-03-18, 16:31
+
Jason Rosenberg 2013-03-18, 20:38
+
Neha Narkhede 2013-03-18, 21:29
+
Jason Rosenberg 2013-03-19, 01:12
Copy link to this message
-
Re: auto-delete empty, old topics?
So, looks like in 0.7, the way to correctly delete topics from the brokers
and zookeeper is as follows -

1. Shutdown producers, consumers and brokers.
2. Delete the topic logs from the brokers
3. Delete the /brokers/topics/<topic> nodes from zookeeper
4. Restart the brokers
5. Restart producers and consumers.

Thanks,
Neha
On Mon, Mar 18, 2013 at 9:31 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Jason,
>
> This is mainly a problem that we don't have a formal way of deleting a
> topic in 0.7, which we are trying to fix in 0.8.
>
> The extra watchers on those deleted topics are likely registered by the
> producers. They should be gone once /brokers/topics/deletedtopic are
> removed from ZK.
>
> You probably don't need to clean /consumers/<appname>/owners.
>
> Thanks,
>
> Jun
>
>
> On Sun, Mar 17, 2013 at 11:11 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
>
> > Jun,
> >
> > There are indeed no nodes under /brokers/topics/deletedtopic
> >
> > Also, do I need to remove the deleted apps from the
> > /consumers/<appname>/owners path?
> >
> > So, should this be considered a bug, or not?  Essentially, each broker
> > knows what topics it has, but the info seems to have not propagated to zk
> > at all?  Is that right?
> >
> > Although from the telnet interface, the removed topics don't show up
> under
> > a 'dump' command, but they do under wchc, etc.  What's that about?
> >
> > Should the original procedure have been to: (1) stop all consumers, then
> > (2) remove the topic files on the brokers, then (3) restart the brokers,
> > then (4) restart the consumers?
> >
> > The consumers have been up and down since the topic files were removed,
> but
> > just not ever at the same time the brokers were down....
> >
> > Jason
> >
> >
> > On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Jason,
> > >
> > > Use zkCli.sh to see if there is any child node in ZK under
> > > /brokers/topics/deletedtopic. You should see none since you already
> > removed
> > > those topics from brokers' log. If so, you can manually remove those
> > paths
> > > in ZK using zkCli.sh (stop the consumers first). After the consumers
> > > restart, you should not see the the deleted topics in the log of the
> > > consumers any more.
> > >
> > > consumer.ZookeeperConsumerConnector -
> > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics
> to
> > > consume = List(<a bazillion topics, most of which have been deleted
> > listed
> > > here>)
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Hi Jun, yes, we're using zk-based producers.  Producers have all been
> > > > restarted (they get cycled many times a day as well).
> > > >
> > > > The specific producers that generated all those bogus topics have
> > > > definitely been restarted, etc.
> > > >
> > > > Using zkCli.sh, I do see all the deleted topics, e.g.:
> > > >
> > > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
> > > > [<1000's of deleted topics>,<also all the remaining valid topics>]
> > > >
> > > > I see also all the deleted topics under /consumers/<appname>/owners/
> > > >
> > > > Jason
> > > >
> > > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Are you using ZK-based producer? If so, those watches could be left
> > by
> > > > the
> > > > > producers if they haven't been restarted. Could you also use
> zkCli.sh
> > > to
> > > > > see if deleted topics are there in ZK?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > So, I connected to zookeeper just using telnet, and using the 4
> > > letter
> > > > > > commands.
> > > > > >
> > > > > > If I do a dump:
> > > > > >
> > > > > > I do not see anything but valid topics, and valid consumer/owners
> > > > > mappings.
> > > > > >

 
+
Jason Rosenberg 2013-03-18, 20:35
+
Jason Rosenberg 2013-03-08, 20:04