Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> auto-delete empty, old topics?


+
Jason Rosenberg 2013-03-06, 23:15
+
Neha Narkhede 2013-03-06, 23:39
+
Jason Rosenberg 2013-03-07, 00:21
+
Neha Narkhede 2013-03-07, 02:08
+
Jason Rosenberg 2013-03-08, 05:13
+
Neha Narkhede 2013-03-08, 06:20
+
Jason Rosenberg 2013-03-14, 21:04
+
Jason Rosenberg 2013-03-14, 21:17
+
Neha Narkhede 2013-03-14, 21:39
+
Jason Rosenberg 2013-03-14, 22:20
+
Neha Narkhede 2013-03-14, 22:23
+
Jun Rao 2013-03-15, 15:18
+
Jason Rosenberg 2013-03-15, 21:19
+
Jun Rao 2013-03-16, 04:48
+
Jason Rosenberg 2013-03-17, 06:08
+
Jun Rao 2013-03-17, 16:09
+
Jason Rosenberg 2013-03-18, 06:11
+
Jun Rao 2013-03-18, 16:31
+
Jason Rosenberg 2013-03-18, 20:38
+
Neha Narkhede 2013-03-18, 21:29
Copy link to this message
-
Re: auto-delete empty, old topics?
One other clue, is that although those topics still show up under
/brokers/topics/<topicname>/ they contain empty sub-nodes.....So zk knows
they don't exist on any broker.  So maybe that's the issue.  Brokers are
successfully removing themselves as serving a topic, but the topic itself
remains.  The consumers then just see the topics (even with no brokers
listed) and continue to watch the topic.

Jason

On Mon, Mar 18, 2013 at 1:37 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Thanks Jun,
>
> I have nothing listed under /brokers/topics/deletedtopic, etc....so that
> doesn't appear to be the issue.
>
> I will try removing now the unwanted topics under /brokers/topics/.
>
> Jason
>
>
> On Mon, Mar 18, 2013 at 9:31 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
>> Jason,
>>
>> This is mainly a problem that we don't have a formal way of deleting a
>> topic in 0.7, which we are trying to fix in 0.8.
>>
>> The extra watchers on those deleted topics are likely registered by the
>> producers. They should be gone once /brokers/topics/deletedtopic are
>> removed from ZK.
>>
>> You probably don't need to clean /consumers/<appname>/owners.
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Sun, Mar 17, 2013 at 11:11 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Jun,
>> >
>> > There are indeed no nodes under /brokers/topics/deletedtopic
>> >
>> > Also, do I need to remove the deleted apps from the
>> > /consumers/<appname>/owners path?
>> >
>> > So, should this be considered a bug, or not?  Essentially, each broker
>> > knows what topics it has, but the info seems to have not propagated to
>> zk
>> > at all?  Is that right?
>> >
>> > Although from the telnet interface, the removed topics don't show up
>> under
>> > a 'dump' command, but they do under wchc, etc.  What's that about?
>> >
>> > Should the original procedure have been to: (1) stop all consumers, then
>> > (2) remove the topic files on the brokers, then (3) restart the brokers,
>> > then (4) restart the consumers?
>> >
>> > The consumers have been up and down since the topic files were removed,
>> but
>> > just not ever at the same time the brokers were down....
>> >
>> > Jason
>> >
>> >
>> > On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> >
>> > > Jason,
>> > >
>> > > Use zkCli.sh to see if there is any child node in ZK under
>> > > /brokers/topics/deletedtopic. You should see none since you already
>> > removed
>> > > those topics from brokers' log. If so, you can manually remove those
>> > paths
>> > > in ZK using zkCli.sh (stop the consumers first). After the consumers
>> > > restart, you should not see the the deleted topics in the log of the
>> > > consumers any more.
>> > >
>> > > consumer.ZookeeperConsumerConnector -
>> > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb
>> Topics to
>> > > consume = List(<a bazillion topics, most of which have been deleted
>> > listed
>> > > here>)
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> > > wrote:
>> > >
>> > > > Hi Jun, yes, we're using zk-based producers.  Producers have all
>> been
>> > > > restarted (they get cycled many times a day as well).
>> > > >
>> > > > The specific producers that generated all those bogus topics have
>> > > > definitely been restarted, etc.
>> > > >
>> > > > Using zkCli.sh, I do see all the deleted topics, e.g.:
>> > > >
>> > > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
>> > > > [<1000's of deleted topics>,<also all the remaining valid topics>]
>> > > >
>> > > > I see also all the deleted topics under /consumers/<appname>/owners/
>> > > >
>> > > > Jason
>> > > >
>> > > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> > > >
>> > > > > Are you using ZK-based producer? If so, those watches could be
>> left
>> > by
>> > > > the
>> > > > > producers if they haven't been restarted. Could you also use
>> zkCli.sh
>> > > to
>> > > > > see if deleted topics are there in ZK?

 
+
Neha Narkhede 2013-03-18, 18:05
+
Jason Rosenberg 2013-03-18, 20:35
+
Jason Rosenberg 2013-03-08, 20:04