Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> auto-delete empty, old topics?


+
Jason Rosenberg 2013-03-06, 23:15
+
Neha Narkhede 2013-03-06, 23:39
+
Jason Rosenberg 2013-03-07, 00:21
+
Neha Narkhede 2013-03-07, 02:08
+
Jason Rosenberg 2013-03-08, 05:13
+
Neha Narkhede 2013-03-08, 06:20
+
Jason Rosenberg 2013-03-14, 21:04
+
Jason Rosenberg 2013-03-14, 21:17
+
Neha Narkhede 2013-03-14, 21:39
+
Jason Rosenberg 2013-03-14, 22:20
+
Neha Narkhede 2013-03-14, 22:23
+
Jun Rao 2013-03-15, 15:18
+
Jason Rosenberg 2013-03-15, 21:19
+
Jun Rao 2013-03-16, 04:48
+
Jason Rosenberg 2013-03-17, 06:08
+
Jun Rao 2013-03-17, 16:09
+
Jason Rosenberg 2013-03-18, 06:11
+
Jun Rao 2013-03-18, 16:31
+
Jason Rosenberg 2013-03-18, 20:38
+
Neha Narkhede 2013-03-18, 21:29
+
Jason Rosenberg 2013-03-19, 01:12
+
Neha Narkhede 2013-03-18, 18:05
Copy link to this message
-
Re: auto-delete empty, old topics?
Ok,

So we'll leave it as this (I won't file a bug).  It is of course less than
ideal to require everything to be brought down when deleting topics!  Since
we consider the cluster to be a high-availability resource.  I assume in
0.8, it won't be necessary to bring everything down to delete a topic?

Jason

On Mon, Mar 18, 2013 at 11:05 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> So, looks like in 0.7, the way to correctly delete topics from the brokers
> and zookeeper is as follows -
>
> 1. Shutdown producers, consumers and brokers.
> 2. Delete the topic logs from the brokers
> 3. Delete the /brokers/topics/<topic> nodes from zookeeper
> 4. Restart the brokers
> 5. Restart producers and consumers.
>
> Thanks,
> Neha
>
>
> On Mon, Mar 18, 2013 at 9:31 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Jason,
> >
> > This is mainly a problem that we don't have a formal way of deleting a
> > topic in 0.7, which we are trying to fix in 0.8.
> >
> > The extra watchers on those deleted topics are likely registered by the
> > producers. They should be gone once /brokers/topics/deletedtopic are
> > removed from ZK.
> >
> > You probably don't need to clean /consumers/<appname>/owners.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Sun, Mar 17, 2013 at 11:11 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Jun,
> > >
> > > There are indeed no nodes under /brokers/topics/deletedtopic
> > >
> > > Also, do I need to remove the deleted apps from the
> > > /consumers/<appname>/owners path?
> > >
> > > So, should this be considered a bug, or not?  Essentially, each broker
> > > knows what topics it has, but the info seems to have not propagated to
> zk
> > > at all?  Is that right?
> > >
> > > Although from the telnet interface, the removed topics don't show up
> > under
> > > a 'dump' command, but they do under wchc, etc.  What's that about?
> > >
> > > Should the original procedure have been to: (1) stop all consumers,
> then
> > > (2) remove the topic files on the brokers, then (3) restart the
> brokers,
> > > then (4) restart the consumers?
> > >
> > > The consumers have been up and down since the topic files were removed,
> > but
> > > just not ever at the same time the brokers were down....
> > >
> > > Jason
> > >
> > >
> > > On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > > > Jason,
> > > >
> > > > Use zkCli.sh to see if there is any child node in ZK under
> > > > /brokers/topics/deletedtopic. You should see none since you already
> > > removed
> > > > those topics from brokers' log. If so, you can manually remove those
> > > paths
> > > > in ZK using zkCli.sh (stop the consumers first). After the consumers
> > > > restart, you should not see the the deleted topics in the log of the
> > > > consumers any more.
> > > >
> > > > consumer.ZookeeperConsumerConnector -
> > > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb
> Topics
> > to
> > > > consume = List(<a bazillion topics, most of which have been deleted
> > > listed
> > > > here>)
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > > wrote:
> > > >
> > > > > Hi Jun, yes, we're using zk-based producers.  Producers have all
> been
> > > > > restarted (they get cycled many times a day as well).
> > > > >
> > > > > The specific producers that generated all those bogus topics have
> > > > > definitely been restarted, etc.
> > > > >
> > > > > Using zkCli.sh, I do see all the deleted topics, e.g.:
> > > > >
> > > > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
> > > > > [<1000's of deleted topics>,<also all the remaining valid topics>]
> > > > >
> > > > > I see also all the deleted topics under
> /consumers/<appname>/owners/
> > > > >
> > > > > Jason
> > > > >
> > > > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > Are you using ZK-based producer? If so, those watches could be
> left
> > > by
> > > > > the

 
+
Jason Rosenberg 2013-03-08, 20:04