We plan to include support for deleting topics before the official release.
The JIRA tracking this feature is
https://issues.apache.org/jira/browse/KAFKA-330

This is mainly an admin tool and is proposed to be run manually. I think
your suggestion is to set some sort of expiration time on topics so when
all data gets garbage collected and the expiration time lapses, the topic
is automatically deleted.

For this expiration to work, we need some mechanism of knowing the size of
the data logs for that topic on all brokers. And only when its empty for
all partitions of that topic, we delete it. This requires some API that
will expose the data size of a topic on a broker, which we don't have right
now. It might be worth thinking about it. Do you mind posting your
suggestions for this tool on that JIRA ?

Thanks,
Neha
On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> I suspect this is not currently supported, but it seems to be for us a real
> use case.
>
> If we have a topic that is no longer receiving messages, and all messages
> have been removed from the brokers, after the log_retention_hours has
> expired, I'd love to have it then automatically remove the topic from
> kafka.
>
> If subsequently a new message arrives for that topic, it could just
> recreate the topic as originally, and continue on.
>
> Is this easy to implement?
> Has it been requested?
> Does it make sense?
> Should I file a jira?
>
> Currently, we had a test client that inadvertently created 1000's of
> topics.  The job of deleting all these appears to not be a cumbersome
> manual process (are there any quick links to the recommended way for doing
> this?).
>
> I am using 0.7.2, but will eventually upgrade to 0.8, once it's released to
> beta....
>
> Thanks,
>
> Jason
>

 

Thanks Neha,

I added a comment to that ticket.

Another question, I'm not sure if the behavior is different in 0.8 from
0.7.  But currently, I'm not predeclaring topics, before sending messages
to topics.  The topics get automagically created if they don't exist.  Will
that still exist in 0.8?  If so, it should also be automatic (using a
default expiration for all such topics), that they automagically go away
after no longer being in use.

What's the procedure for deleting topics in kafka 0.7.2?

Jason

On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> We plan to include support for deleting topics before the official release.
> The JIRA tracking this feature is
> https://issues.apache.org/jira/browse/KAFKA-330
>
> This is mainly an admin tool and is proposed to be run manually. I think
> your suggestion is to set some sort of expiration time on topics so when
> all data gets garbage collected and the expiration time lapses, the topic
> is automatically deleted.
>
> For this expiration to work, we need some mechanism of knowing the size of
> the data logs for that topic on all brokers. And only when its empty for
> all partitions of that topic, we delete it. This requires some API that
> will expose the data size of a topic on a broker, which we don't have right
> now. It might be worth thinking about it. Do you mind posting your
> suggestions for this tool on that JIRA ?
>
> Thanks,
> Neha
>
>
> On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > I suspect this is not currently supported, but it seems to be for us a
> real
> > use case.
> >
> > If we have a topic that is no longer receiving messages, and all messages
> > have been removed from the brokers, after the log_retention_hours has
> > expired, I'd love to have it then automatically remove the topic from
> > kafka.
> >
> > If subsequently a new message arrives for that topic, it could just
> > recreate the topic as originally, and continue on.
> >
> > Is this easy to implement?
> > Has it been requested?
> > Does it make sense?
> > Should I file a jira?
> >
> > Currently, we had a test client that inadvertently created 1000's of
> > topics.  The job of deleting all these appears to not be a cumbersome
> > manual process (are there any quick links to the recommended way for
> doing
> > this?).
> >
> > I am using 0.7.2, but will eventually upgrade to 0.8, once it's released
> to
> > beta....
> >
> > Thanks,
> >
> > Jason
> >
>

 

>> The topics get automagically created if they don't exist.  Will
that still exist in 0.8?

In 0.8, this is controlled by the auto.create.topics.enable config on the
brokers. If this is set to true, topics will be created when a topic
metadata request is sent for a new topic. This feature is provided to aid
with the 0.7->0.8 migration tool and the mirror maker tools. These tools
copy data for almost all topics from one cluster to the other, so auto
creating topics makes it much easier to migrate data between clusters.

>> What's the procedure for deleting topics in kafka 0.7.2?

There is no official way to delete a topic. You have to delete the data
directory and bounce the brokers to take note of that.

Thanks,
Neha
On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Thanks Neha,
>
> I added a comment to that ticket.
>
> Another question, I'm not sure if the behavior is different in 0.8 from
> 0.7.  But currently, I'm not predeclaring topics, before sending messages
> to topics.  The topics get automagically created if they don't exist.  Will
> that still exist in 0.8?  If so, it should also be automatic (using a
> default expiration for all such topics), that they automagically go away
> after no longer being in use.
>
> What's the procedure for deleting topics in kafka 0.7.2?
>
> Jason
>
> On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > We plan to include support for deleting topics before the official
> release.
> > The JIRA tracking this feature is
> > https://issues.apache.org/jira/browse/KAFKA-330
> >
> > This is mainly an admin tool and is proposed to be run manually. I think
> > your suggestion is to set some sort of expiration time on topics so when
> > all data gets garbage collected and the expiration time lapses, the topic
> > is automatically deleted.
> >
> > For this expiration to work, we need some mechanism of knowing the size
> of
> > the data logs for that topic on all brokers. And only when its empty for
> > all partitions of that topic, we delete it. This requires some API that
> > will expose the data size of a topic on a broker, which we don't have
> right
> > now. It might be worth thinking about it. Do you mind posting your
> > suggestions for this tool on that JIRA ?
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > I suspect this is not currently supported, but it seems to be for us a
> > real
> > > use case.
> > >
> > > If we have a topic that is no longer receiving messages, and all
> messages
> > > have been removed from the brokers, after the log_retention_hours has
> > > expired, I'd love to have it then automatically remove the topic from
> > > kafka.
> > >
> > > If subsequently a new message arrives for that topic, it could just
> > > recreate the topic as originally, and continue on.
> > >
> > > Is this easy to implement?
> > > Has it been requested?
> > > Does it make sense?
> > > Should I file a jira?
> > >
> > > Currently, we had a test client that inadvertently created 1000's of
> > > topics.  The job of deleting all these appears to not be a cumbersome
> > > manual process (are there any quick links to the recommended way for
> > doing
> > > this?).
> > >
> > > I am using 0.7.2, but will eventually upgrade to 0.8, once it's
> released
> > to
> > > beta....
> > >
> > > Thanks,
> > >
> > > Jason
> > >
> >
>

 

Thanks Neha,

So are you saying that on 0.7.2, to delete a topic I need only remove it's
data log directory from each broker, and the restart the brokers?  Is it ok
if it's a rolling restart?

For some reason I thought I also had to do something on zookeeper servers
too, to remove the topics.  Is that not true?

Jason

On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> >> The topics get automagically created if they don't exist.  Will
> that still exist in 0.8?
>
> In 0.8, this is controlled by the auto.create.topics.enable config on the
> brokers. If this is set to true, topics will be created when a topic
> metadata request is sent for a new topic. This feature is provided to aid
> with the 0.7->0.8 migration tool and the mirror maker tools. These tools
> copy data for almost all topics from one cluster to the other, so auto
> creating topics makes it much easier to migrate data between clusters.
>
> >> What's the procedure for deleting topics in kafka 0.7.2?
>
> There is no official way to delete a topic. You have to delete the data
> directory and bounce the brokers to take note of that.
>
> Thanks,
> Neha
>
>
> On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Thanks Neha,
> >
> > I added a comment to that ticket.
> >
> > Another question, I'm not sure if the behavior is different in 0.8 from
> > 0.7.  But currently, I'm not predeclaring topics, before sending messages
> > to topics.  The topics get automagically created if they don't exist.
>  Will
> > that still exist in 0.8?  If so, it should also be automatic (using a
> > default expiration for all such topics), that they automagically go away
> > after no longer being in use.
> >
> > What's the procedure for deleting topics in kafka 0.7.2?
> >
> > Jason
> >
> > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > We plan to include support for deleting topics before the official
> > release.
> > > The JIRA tracking this feature is
> > > https://issues.apache.org/jira/browse/KAFKA-330
> > >
> > > This is mainly an admin tool and is proposed to be run manually. I
> think
> > > your suggestion is to set some sort of expiration time on topics so
> when
> > > all data gets garbage collected and the expiration time lapses, the
> topic
> > > is automatically deleted.
> > >
> > > For this expiration to work, we need some mechanism of knowing the size
> > of
> > > the data logs for that topic on all brokers. And only when its empty
> for
> > > all partitions of that topic, we delete it. This requires some API that
> > > will expose the data size of a topic on a broker, which we don't have
> > right
> > > now. It might be worth thinking about it. Do you mind posting your
> > > suggestions for this tool on that JIRA ?
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > I suspect this is not currently supported, but it seems to be for us
> a
> > > real
> > > > use case.
> > > >
> > > > If we have a topic that is no longer receiving messages, and all
> > messages
> > > > have been removed from the brokers, after the log_retention_hours has
> > > > expired, I'd love to have it then automatically remove the topic from
> > > > kafka.
> > > >
> > > > If subsequently a new message arrives for that topic, it could just
> > > > recreate the topic as originally, and continue on.
> > > >
> > > > Is this easy to implement?
> > > > Has it been requested?
> > > > Does it make sense?
> > > > Should I file a jira?
> > > >
> > > > Currently, we had a test client that inadvertently created 1000's of
> > > > topics.  The job of deleting all these appears to not be a cumbersome
> > > > manual process (are there any quick links to the recommended way for
> > > doing
> > > > this?).
> > > >
> > > > I am using 0.7.2, but will eventually upgrade to 0.8, once it's
> > released
> > > to
> > > > beta....
> > > >
> > >
 

In Kafka 0.7.2, data log directory delete and then rolling restart will
work to get rid of the topic completely from zookeeper as well as the
broker.
This is because the broker registers the topics and partitions from its
data directory into zookeeper on startup.

Thanks,
Neha
On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Thanks Neha,
>
> So are you saying that on 0.7.2, to delete a topic I need only remove it's
> data log directory from each broker, and the restart the brokers?  Is it ok
> if it's a rolling restart?
>
> For some reason I thought I also had to do something on zookeeper servers
> too, to remove the topics.  Is that not true?
>
> Jason
>
> On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > >> The topics get automagically created if they don't exist.  Will
> > that still exist in 0.8?
> >
> > In 0.8, this is controlled by the auto.create.topics.enable config on the
> > brokers. If this is set to true, topics will be created when a topic
> > metadata request is sent for a new topic. This feature is provided to aid
> > with the 0.7->0.8 migration tool and the mirror maker tools. These tools
> > copy data for almost all topics from one cluster to the other, so auto
> > creating topics makes it much easier to migrate data between clusters.
> >
> > >> What's the procedure for deleting topics in kafka 0.7.2?
> >
> > There is no official way to delete a topic. You have to delete the data
> > directory and bounce the brokers to take note of that.
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Thanks Neha,
> > >
> > > I added a comment to that ticket.
> > >
> > > Another question, I'm not sure if the behavior is different in 0.8 from
> > > 0.7.  But currently, I'm not predeclaring topics, before sending
> messages
> > > to topics.  The topics get automagically created if they don't exist.
> >  Will
> > > that still exist in 0.8?  If so, it should also be automatic (using a
> > > default expiration for all such topics), that they automagically go
> away
> > > after no longer being in use.
> > >
> > > What's the procedure for deleting topics in kafka 0.7.2?
> > >
> > > Jason
> > >
> > > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > We plan to include support for deleting topics before the official
> > > release.
> > > > The JIRA tracking this feature is
> > > > https://issues.apache.org/jira/browse/KAFKA-330
> > > >
> > > > This is mainly an admin tool and is proposed to be run manually. I
> > think
> > > > your suggestion is to set some sort of expiration time on topics so
> > when
> > > > all data gets garbage collected and the expiration time lapses, the
> > topic
> > > > is automatically deleted.
> > > >
> > > > For this expiration to work, we need some mechanism of knowing the
> size
> > > of
> > > > the data logs for that topic on all brokers. And only when its empty
> > for
> > > > all partitions of that topic, we delete it. This requires some API
> that
> > > > will expose the data size of a topic on a broker, which we don't have
> > > right
> > > > now. It might be worth thinking about it. Do you mind posting your
> > > > suggestions for this tool on that JIRA ?
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > I suspect this is not currently supported, but it seems to be for
> us
> > a
> > > > real
> > > > > use case.
> > > > >
> > > > > If we have a topic that is no longer receiving messages, and all
> > > messages
> > > > > have been removed from the brokers, after the log_retention_hours
> has
> > > > > expired, I'd love to have it then automatically remove the topic
> from
> > > > > kafka.
> > > > >
> > > > > If subsequently a new message arrives for that topic, it could just
> > > > > recreate the topic as originally, and continue on.

 

Hi Neha,

So I did this, but I still see the full list of topics (most of which have
been deleted), in the consumer logs, e.g.:

consumer.ZookeeperConsumerConnector -
samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
consume = List(<a bazillion topics, most of which have been deleted listed
here>)

I select topics using the white list topic selector (using a wild-card
pattern, etc.).

Thoughts?

Does zookeeper keep separate info for the consumer, that needs to be
removed?

Jason

On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> In Kafka 0.7.2, data log directory delete and then rolling restart will
> work to get rid of the topic completely from zookeeper as well as the
> broker.
> This is because the broker registers the topics and partitions from its
> data directory into zookeeper on startup.
>
> Thanks,
> Neha
>
>
> On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Thanks Neha,
> >
> > So are you saying that on 0.7.2, to delete a topic I need only remove
> it's
> > data log directory from each broker, and the restart the brokers?  Is it
> ok
> > if it's a rolling restart?
> >
> > For some reason I thought I also had to do something on zookeeper servers
> > too, to remove the topics.  Is that not true?
> >
> > Jason
> >
> > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > >> The topics get automagically created if they don't exist.  Will
> > > that still exist in 0.8?
> > >
> > > In 0.8, this is controlled by the auto.create.topics.enable config on
> the
> > > brokers. If this is set to true, topics will be created when a topic
> > > metadata request is sent for a new topic. This feature is provided to
> aid
> > > with the 0.7->0.8 migration tool and the mirror maker tools. These
> tools
> > > copy data for almost all topics from one cluster to the other, so auto
> > > creating topics makes it much easier to migrate data between clusters.
> > >
> > > >> What's the procedure for deleting topics in kafka 0.7.2?
> > >
> > > There is no official way to delete a topic. You have to delete the data
> > > directory and bounce the brokers to take note of that.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Thanks Neha,
> > > >
> > > > I added a comment to that ticket.
> > > >
> > > > Another question, I'm not sure if the behavior is different in 0.8
> from
> > > > 0.7.  But currently, I'm not predeclaring topics, before sending
> > messages
> > > > to topics.  The topics get automagically created if they don't exist.
> > >  Will
> > > > that still exist in 0.8?  If so, it should also be automatic (using a
> > > > default expiration for all such topics), that they automagically go
> > away
> > > > after no longer being in use.
> > > >
> > > > What's the procedure for deleting topics in kafka 0.7.2?
> > > >
> > > > Jason
> > > >
> > > > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > We plan to include support for deleting topics before the official
> > > > release.
> > > > > The JIRA tracking this feature is
> > > > > https://issues.apache.org/jira/browse/KAFKA-330
> > > > >
> > > > > This is mainly an admin tool and is proposed to be run manually. I
> > > think
> > > > > your suggestion is to set some sort of expiration time on topics so
> > > when
> > > > > all data gets garbage collected and the expiration time lapses, the
> > > topic
> > > > > is automatically deleted.
> > > > >
> > > > > For this expiration to work, we need some mechanism of knowing the
> > size
> > > > of
> > > > > the data logs for that topic on all brokers. And only when its
> empty
> > > for
> > > > > all partitions of that topic, we delete it. This requires some API
> > that
> > > > > will expose the data size of a topic on a broker, which we don't
> have
> > > > right
> > > > > now. It might be worth thinking about it. Do you mind posting your

 

Also,

I see a bazillion consecutive log lines like this:

2013-03-14 19:54:13,306  INFO [Thread-4] consumer.ConsumerIterator -
Clearing the current data chunk for this consumer iterator

With the same content (not sure how useful that is!).

Jason

On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Hi Neha,
>
> So I did this, but I still see the full list of topics (most of which have
> been deleted), in the consumer logs, e.g.:
>
> consumer.ZookeeperConsumerConnector -
> samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
> consume = List(<a bazillion topics, most of which have been deleted listed
> here>)
>
> I select topics using the white list topic selector (using a wild-card
> pattern, etc.).
>
> Thoughts?
>
> Does zookeeper keep separate info for the consumer, that needs to be
> removed?
>
> Jason
>
> On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:
>
>> In Kafka 0.7.2, data log directory delete and then rolling restart will
>> work to get rid of the topic completely from zookeeper as well as the
>> broker.
>> This is because the broker registers the topics and partitions from its
>> data directory into zookeeper on startup.
>>
>> Thanks,
>> Neha
>>
>>
>> On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>>
>> > Thanks Neha,
>> >
>> > So are you saying that on 0.7.2, to delete a topic I need only remove
>> it's
>> > data log directory from each broker, and the restart the brokers?  Is
>> it ok
>> > if it's a rolling restart?
>> >
>> > For some reason I thought I also had to do something on zookeeper
>> servers
>> > too, to remove the topics.  Is that not true?
>> >
>> > Jason
>> >
>> > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <[EMAIL PROTECTED]
>> > >wrote:
>> >
>> > > >> The topics get automagically created if they don't exist.  Will
>> > > that still exist in 0.8?
>> > >
>> > > In 0.8, this is controlled by the auto.create.topics.enable config on
>> the
>> > > brokers. If this is set to true, topics will be created when a topic
>> > > metadata request is sent for a new topic. This feature is provided to
>> aid
>> > > with the 0.7->0.8 migration tool and the mirror maker tools. These
>> tools
>> > > copy data for almost all topics from one cluster to the other, so auto
>> > > creating topics makes it much easier to migrate data between clusters.
>> > >
>> > > >> What's the procedure for deleting topics in kafka 0.7.2?
>> > >
>> > > There is no official way to delete a topic. You have to delete the
>> data
>> > > directory and bounce the brokers to take note of that.
>> > >
>> > > Thanks,
>> > > Neha
>> > >
>> > >
>> > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> > wrote:
>> > >
>> > > > Thanks Neha,
>> > > >
>> > > > I added a comment to that ticket.
>> > > >
>> > > > Another question, I'm not sure if the behavior is different in 0.8
>> from
>> > > > 0.7.  But currently, I'm not predeclaring topics, before sending
>> > messages
>> > > > to topics.  The topics get automagically created if they don't
>> exist.
>> > >  Will
>> > > > that still exist in 0.8?  If so, it should also be automatic (using
>> a
>> > > > default expiration for all such topics), that they automagically go
>> > away
>> > > > after no longer being in use.
>> > > >
>> > > > What's the procedure for deleting topics in kafka 0.7.2?
>> > > >
>> > > > Jason
>> > > >
>> > > > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <
>> [EMAIL PROTECTED]
>> > > > >wrote:
>> > > >
>> > > > > We plan to include support for deleting topics before the official
>> > > > release.
>> > > > > The JIRA tracking this feature is
>> > > > > https://issues.apache.org/jira/browse/KAFKA-330
>> > > > >
>> > > > > This is mainly an admin tool and is proposed to be run manually. I
>> > > think
>> > > > > your suggestion is to set some sort of expiration time on topics
>> so
>> > > when
>> > > > > all data gets garbage collected and the expiration time lapses,

 

Jason,

Let me see if I understood what you did here. In Kafka 0.7.2, you deleted
the Kafka log files from the server and bounced the broker. This should've
ideally deleted those topics from zookeeper, the consumer reads the same
zookeeper paths that the broker writes. Doing this should cause rebalancing
on the consumer and it should readjust its consumption to include the
remaining topics. But looks like that didn't happen somehow ?

I agree that log line is redundant, I think it is removed in 0.8

Thanks,
Neha
On Thu, Mar 14, 2013 at 2:17 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Also,
>
> I see a bazillion consecutive log lines like this:
>
> 2013-03-14 19:54:13,306  INFO [Thread-4] consumer.ConsumerIterator -
> Clearing the current data chunk for this consumer iterator
>
> With the same content (not sure how useful that is!).
>
> Jason
>
> On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Hi Neha,
> >
> > So I did this, but I still see the full list of topics (most of which
> have
> > been deleted), in the consumer logs, e.g.:
> >
> > consumer.ZookeeperConsumerConnector -
> > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
> > consume = List(<a bazillion topics, most of which have been deleted
> listed
> > here>)
> >
> > I select topics using the white list topic selector (using a wild-card
> > pattern, etc.).
> >
> > Thoughts?
> >
> > Does zookeeper keep separate info for the consumer, that needs to be
> > removed?
> >
> > Jason
> >
> > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
> >
> >> In Kafka 0.7.2, data log directory delete and then rolling restart will
> >> work to get rid of the topic completely from zookeeper as well as the
> >> broker.
> >> This is because the broker registers the topics and partitions from its
> >> data directory into zookeeper on startup.
> >>
> >> Thanks,
> >> Neha
> >>
> >>
> >> On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >>
> >> > Thanks Neha,
> >> >
> >> > So are you saying that on 0.7.2, to delete a topic I need only remove
> >> it's
> >> > data log directory from each broker, and the restart the brokers?  Is
> >> it ok
> >> > if it's a rolling restart?
> >> >
> >> > For some reason I thought I also had to do something on zookeeper
> >> servers
> >> > too, to remove the topics.  Is that not true?
> >> >
> >> > Jason
> >> >
> >> > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> >> > >wrote:
> >> >
> >> > > >> The topics get automagically created if they don't exist.  Will
> >> > > that still exist in 0.8?
> >> > >
> >> > > In 0.8, this is controlled by the auto.create.topics.enable config
> on
> >> the
> >> > > brokers. If this is set to true, topics will be created when a topic
> >> > > metadata request is sent for a new topic. This feature is provided
> to
> >> aid
> >> > > with the 0.7->0.8 migration tool and the mirror maker tools. These
> >> tools
> >> > > copy data for almost all topics from one cluster to the other, so
> auto
> >> > > creating topics makes it much easier to migrate data between
> clusters.
> >> > >
> >> > > >> What's the procedure for deleting topics in kafka 0.7.2?
> >> > >
> >> > > There is no official way to delete a topic. You have to delete the
> >> data
> >> > > directory and bounce the brokers to take note of that.
> >> > >
> >> > > Thanks,
> >> > > Neha
> >> > >
> >> > >
> >> > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> >> > wrote:
> >> > >
> >> > > > Thanks Neha,
> >> > > >
> >> > > > I added a comment to that ticket.
> >> > > >
> >> > > > Another question, I'm not sure if the behavior is different in 0.8
> >> from
> >> > > > 0.7.  But currently, I'm not predeclaring topics, before sending
> >> > messages
> >> > > > to topics.  The topics get automagically created if they don't
> >> exist.
> >> > >  Will
> >> > > > that still exist in 0.8?  If so, it should also be automatic

 

Yes, you're description matches what I did.  And the brokers have been
bounced many times since then (they are auto-deployed many times a day,
etc.).  And the consumers also have been restarted many times since then.

Could it be related to using the white-list topic selector, etc.?

Let me know if you need me to look a the state of my zookeeper data....

Jason

On Thu, Mar 14, 2013 at 2:39 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Jason,
>
> Let me see if I understood what you did here. In Kafka 0.7.2, you deleted
> the Kafka log files from the server and bounced the broker. This should've
> ideally deleted those topics from zookeeper, the consumer reads the same
> zookeeper paths that the broker writes. Doing this should cause rebalancing
> on the consumer and it should readjust its consumption to include the
> remaining topics. But looks like that didn't happen somehow ?
>
> I agree that log line is redundant, I think it is removed in 0.8
>
> Thanks,
> Neha
>
>
> On Thu, Mar 14, 2013 at 2:17 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Also,
> >
> > I see a bazillion consecutive log lines like this:
> >
> > 2013-03-14 19:54:13,306  INFO [Thread-4] consumer.ConsumerIterator -
> > Clearing the current data chunk for this consumer iterator
> >
> > With the same content (not sure how useful that is!).
> >
> > Jason
> >
> > On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hi Neha,
> > >
> > > So I did this, but I still see the full list of topics (most of which
> > have
> > > been deleted), in the consumer logs, e.g.:
> > >
> > > consumer.ZookeeperConsumerConnector -
> > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics
> to
> > > consume = List(<a bazillion topics, most of which have been deleted
> > listed
> > > here>)
> > >
> > > I select topics using the white list topic selector (using a wild-card
> > > pattern, etc.).
> > >
> > > Thoughts?
> > >
> > > Does zookeeper keep separate info for the consumer, that needs to be
> > > removed?
> > >
> > > Jason
> > >
> > > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > >wrote:
> > >
> > >> In Kafka 0.7.2, data log directory delete and then rolling restart
> will
> > >> work to get rid of the topic completely from zookeeper as well as the
> > >> broker.
> > >> This is because the broker registers the topics and partitions from
> its
> > >> data directory into zookeeper on startup.
> > >>
> > >> Thanks,
> > >> Neha
> > >>
> > >>
> > >> On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >>
> > >> > Thanks Neha,
> > >> >
> > >> > So are you saying that on 0.7.2, to delete a topic I need only
> remove
> > >> it's
> > >> > data log directory from each broker, and the restart the brokers?
>  Is
> > >> it ok
> > >> > if it's a rolling restart?
> > >> >
> > >> > For some reason I thought I also had to do something on zookeeper
> > >> servers
> > >> > too, to remove the topics.  Is that not true?
> > >> >
> > >> > Jason
> > >> >
> > >> > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > >> > >wrote:
> > >> >
> > >> > > >> The topics get automagically created if they don't exist.  Will
> > >> > > that still exist in 0.8?
> > >> > >
> > >> > > In 0.8, this is controlled by the auto.create.topics.enable config
> > on
> > >> the
> > >> > > brokers. If this is set to true, topics will be created when a
> topic
> > >> > > metadata request is sent for a new topic. This feature is provided
> > to
> > >> aid
> > >> > > with the 0.7->0.8 migration tool and the mirror maker tools. These
> > >> tools
> > >> > > copy data for almost all topics from one cluster to the other, so
> > auto
> > >> > > creating topics makes it much easier to migrate data between
> > clusters.
> > >> > >
> > >> > > >> What's the procedure for deleting topics in kafka 0.7.2?
> > >> > >
> > >> > > There is no official way to delete a topic. You have to delete the

 

This could be a bug with the topic discovery logic in the wildcard
consumer. Please can you file a bug and attach your consumer logs there ?

Thanks,
Neha
On Thu, Mar 14, 2013 at 3:20 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Yes, you're description matches what I did.  And the brokers have been
> bounced many times since then (they are auto-deployed many times a day,
> etc.).  And the consumers also have been restarted many times since then.
>
> Could it be related to using the white-list topic selector, etc.?
>
> Let me know if you need me to look a the state of my zookeeper data....
>
> Jason
>
> On Thu, Mar 14, 2013 at 2:39 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > Jason,
> >
> > Let me see if I understood what you did here. In Kafka 0.7.2, you deleted
> > the Kafka log files from the server and bounced the broker. This
> should've
> > ideally deleted those topics from zookeeper, the consumer reads the same
> > zookeeper paths that the broker writes. Doing this should cause
> rebalancing
> > on the consumer and it should readjust its consumption to include the
> > remaining topics. But looks like that didn't happen somehow ?
> >
> > I agree that log line is redundant, I think it is removed in 0.8
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Mar 14, 2013 at 2:17 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Also,
> > >
> > > I see a bazillion consecutive log lines like this:
> > >
> > > 2013-03-14 19:54:13,306  INFO [Thread-4] consumer.ConsumerIterator -
> > > Clearing the current data chunk for this consumer iterator
> > >
> > > With the same content (not sure how useful that is!).
> > >
> > > Jason
> > >
> > > On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Hi Neha,
> > > >
> > > > So I did this, but I still see the full list of topics (most of which
> > > have
> > > > been deleted), in the consumer logs, e.g.:
> > > >
> > > > consumer.ZookeeperConsumerConnector -
> > > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb
> Topics
> > to
> > > > consume = List(<a bazillion topics, most of which have been deleted
> > > listed
> > > > here>)
> > > >
> > > > I select topics using the white list topic selector (using a
> wild-card
> > > > pattern, etc.).
> > > >
> > > > Thoughts?
> > > >
> > > > Does zookeeper keep separate info for the consumer, that needs to be
> > > > removed?
> > > >
> > > > Jason
> > > >
> > > > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > >wrote:
> > > >
> > > >> In Kafka 0.7.2, data log directory delete and then rolling restart
> > will
> > > >> work to get rid of the topic completely from zookeeper as well as
> the
> > > >> broker.
> > > >> This is because the broker registers the topics and partitions from
> > its
> > > >> data directory into zookeeper on startup.
> > > >>
> > > >> Thanks,
> > > >> Neha
> > > >>
> > > >>
> > > >> On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >>
> > > >> > Thanks Neha,
> > > >> >
> > > >> > So are you saying that on 0.7.2, to delete a topic I need only
> > remove
> > > >> it's
> > > >> > data log directory from each broker, and the restart the brokers?
> >  Is
> > > >> it ok
> > > >> > if it's a rolling restart?
> > > >> >
> > > >> > For some reason I thought I also had to do something on zookeeper
> > > >> servers
> > > >> > too, to remove the topics.  Is that not true?
> > > >> >
> > > >> > Jason
> > > >> >
> > > >> > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <
> > > [EMAIL PROTECTED]
> > > >> > >wrote:
> > > >> >
> > > >> > > >> The topics get automagically created if they don't exist.
>  Will
> > > >> > > that still exist in 0.8?
> > > >> > >
> > > >> > > In 0.8, this is controlled by the auto.create.topics.enable
> config
> > > on
> > > >> the
> > > >> > > brokers. If this is set to true, topics will be created when a
> > topic
> > > >> > > metadata request is sent for a new topic. This feature is

 

Could you check if the following path for a deleted topic exists in ZK? It
should have no children.

/brokers/topics/[topic]

If this is the case, try manually removing those paths from ZK (when the
brokers and the consumers are down).

Thanks,

Jun

On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Hi Neha,
>
> So I did this, but I still see the full list of topics (most of which have
> been deleted), in the consumer logs, e.g.:
>
> consumer.ZookeeperConsumerConnector -
> samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
> consume = List(<a bazillion topics, most of which have been deleted listed
> here>)
>
> I select topics using the white list topic selector (using a wild-card
> pattern, etc.).
>
> Thoughts?
>
> Does zookeeper keep separate info for the consumer, that needs to be
> removed?
>
> Jason
>
> On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > In Kafka 0.7.2, data log directory delete and then rolling restart will
> > work to get rid of the topic completely from zookeeper as well as the
> > broker.
> > This is because the broker registers the topics and partitions from its
> > data directory into zookeeper on startup.
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Thanks Neha,
> > >
> > > So are you saying that on 0.7.2, to delete a topic I need only remove
> > it's
> > > data log directory from each broker, and the restart the brokers?  Is
> it
> > ok
> > > if it's a rolling restart?
> > >
> > > For some reason I thought I also had to do something on zookeeper
> servers
> > > too, to remove the topics.  Is that not true?
> > >
> > > Jason
> > >
> > > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > >> The topics get automagically created if they don't exist.  Will
> > > > that still exist in 0.8?
> > > >
> > > > In 0.8, this is controlled by the auto.create.topics.enable config on
> > the
> > > > brokers. If this is set to true, topics will be created when a topic
> > > > metadata request is sent for a new topic. This feature is provided to
> > aid
> > > > with the 0.7->0.8 migration tool and the mirror maker tools. These
> > tools
> > > > copy data for almost all topics from one cluster to the other, so
> auto
> > > > creating topics makes it much easier to migrate data between
> clusters.
> > > >
> > > > >> What's the procedure for deleting topics in kafka 0.7.2?
> > > >
> > > > There is no official way to delete a topic. You have to delete the
> data
> > > > directory and bounce the brokers to take note of that.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Thanks Neha,
> > > > >
> > > > > I added a comment to that ticket.
> > > > >
> > > > > Another question, I'm not sure if the behavior is different in 0.8
> > from
> > > > > 0.7.  But currently, I'm not predeclaring topics, before sending
> > > messages
> > > > > to topics.  The topics get automagically created if they don't
> exist.
> > > >  Will
> > > > > that still exist in 0.8?  If so, it should also be automatic
> (using a
> > > > > default expiration for all such topics), that they automagically go
> > > away
> > > > > after no longer being in use.
> > > > >
> > > > > What's the procedure for deleting topics in kafka 0.7.2?
> > > > >
> > > > > Jason
> > > > >
> > > > > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > We plan to include support for deleting topics before the
> official
> > > > > release.
> > > > > > The JIRA tracking this feature is
> > > > > > https://issues.apache.org/jira/browse/KAFKA-330
> > > > > >
> > > > > > This is mainly an admin tool and is proposed to be run manually.
> I
> > > > think
> > > > > > your suggestion is to set some sort of expiration time on topics

 

Jun,

So, I connected to zookeeper just using telnet, and using the 4 letter
commands.

If I do a dump:

I do not see anything but valid topics, and valid consumer/owners mappings.

If I check watches, I see all the 1000's of bogus topics, e.c.:

wchc:
....
/brokers/topics/<obsolete-topic>
....

or

wchp
....
/brokers/topics/<obsolete-topic>.
0x383d6fe1b83f0008
0x383d6fe1b83f0009
0x7c3d6f9020d00041
0x833d6fe2826e0000
0x383d6fe1b83f000a
....

The kafka and zookeeper servers, as well as the consumer apps, are rolled
several times a day, automatically.

I'm not sure how to remove the watches, is that what I need to do?

Since this is a highly available system, can I bring down each zk server 1
at a time, and clear the bogus topics (or watches)?  How do I do this?

Jason

On Fri, Mar 15, 2013 at 8:17 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Could you check if the following path for a deleted topic exists in ZK? It
> should have no children.
>
> /brokers/topics/[topic]
>
> If this is the case, try manually removing those paths from ZK (when the
> brokers and the consumers are down).
>
> Thanks,
>
> Jun
>
> On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Hi Neha,
> >
> > So I did this, but I still see the full list of topics (most of which
> have
> > been deleted), in the consumer logs, e.g.:
> >
> > consumer.ZookeeperConsumerConnector -
> > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
> > consume = List(<a bazillion topics, most of which have been deleted
> listed
> > here>)
> >
> > I select topics using the white list topic selector (using a wild-card
> > pattern, etc.).
> >
> > Thoughts?
> >
> > Does zookeeper keep separate info for the consumer, that needs to be
> > removed?
> >
> > Jason
> >
> > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > In Kafka 0.7.2, data log directory delete and then rolling restart will
> > > work to get rid of the topic completely from zookeeper as well as the
> > > broker.
> > > This is because the broker registers the topics and partitions from its
> > > data directory into zookeeper on startup.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Thanks Neha,
> > > >
> > > > So are you saying that on 0.7.2, to delete a topic I need only remove
> > > it's
> > > > data log directory from each broker, and the restart the brokers?  Is
> > it
> > > ok
> > > > if it's a rolling restart?
> > > >
> > > > For some reason I thought I also had to do something on zookeeper
> > servers
> > > > too, to remove the topics.  Is that not true?
> > > >
> > > > Jason
> > > >
> > > > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > >> The topics get automagically created if they don't exist.  Will
> > > > > that still exist in 0.8?
> > > > >
> > > > > In 0.8, this is controlled by the auto.create.topics.enable config
> on
> > > the
> > > > > brokers. If this is set to true, topics will be created when a
> topic
> > > > > metadata request is sent for a new topic. This feature is provided
> to
> > > aid
> > > > > with the 0.7->0.8 migration tool and the mirror maker tools. These
> > > tools
> > > > > copy data for almost all topics from one cluster to the other, so
> > auto
> > > > > creating topics makes it much easier to migrate data between
> > clusters.
> > > > >
> > > > > >> What's the procedure for deleting topics in kafka 0.7.2?
> > > > >
> > > > > There is no official way to delete a topic. You have to delete the
> > data
> > > > > directory and bounce the brokers to take note of that.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >
> > > > > > Thanks Neha,
> > > > > >
> > > > > > I added a comment to that ticket.
> > > > > >
> > > > > > Another question, I'm not sure if the behavior is different in

 

Are you using ZK-based producer? If so, those watches could be left by the
producers if they haven't been restarted. Could you also use zkCli.sh to
see if deleted topics are there in ZK?

Thanks,

Jun

On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Jun,
>
> So, I connected to zookeeper just using telnet, and using the 4 letter
> commands.
>
> If I do a dump:
>
> I do not see anything but valid topics, and valid consumer/owners mappings.
>
> If I check watches, I see all the 1000's of bogus topics, e.c.:
>
> wchc:
> ....
> /brokers/topics/<obsolete-topic>
> ....
>
> or
>
> wchp
> ....
> /brokers/topics/<obsolete-topic>.
> 0x383d6fe1b83f0008
> 0x383d6fe1b83f0009
> 0x7c3d6f9020d00041
> 0x833d6fe2826e0000
> 0x383d6fe1b83f000a
> ....
>
> The kafka and zookeeper servers, as well as the consumer apps, are rolled
> several times a day, automatically.
>
> I'm not sure how to remove the watches, is that what I need to do?
>
> Since this is a highly available system, can I bring down each zk server 1
> at a time, and clear the bogus topics (or watches)?  How do I do this?
>
> Jason
>
>
>
> On Fri, Mar 15, 2013 at 8:17 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Could you check if the following path for a deleted topic exists in ZK?
> It
> > should have no children.
> >
> > /brokers/topics/[topic]
> >
> > If this is the case, try manually removing those paths from ZK (when the
> > brokers and the consumers are down).
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hi Neha,
> > >
> > > So I did this, but I still see the full list of topics (most of which
> > have
> > > been deleted), in the consumer logs, e.g.:
> > >
> > > consumer.ZookeeperConsumerConnector -
> > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics
> to
> > > consume = List(<a bazillion topics, most of which have been deleted
> > listed
> > > here>)
> > >
> > > I select topics using the white list topic selector (using a wild-card
> > > pattern, etc.).
> > >
> > > Thoughts?
> > >
> > > Does zookeeper keep separate info for the consumer, that needs to be
> > > removed?
> > >
> > > Jason
> > >
> > > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > In Kafka 0.7.2, data log directory delete and then rolling restart
> will
> > > > work to get rid of the topic completely from zookeeper as well as the
> > > > broker.
> > > > This is because the broker registers the topics and partitions from
> its
> > > > data directory into zookeeper on startup.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Thanks Neha,
> > > > >
> > > > > So are you saying that on 0.7.2, to delete a topic I need only
> remove
> > > > it's
> > > > > data log directory from each broker, and the restart the brokers?
>  Is
> > > it
> > > > ok
> > > > > if it's a rolling restart?
> > > > >
> > > > > For some reason I thought I also had to do something on zookeeper
> > > servers
> > > > > too, to remove the topics.  Is that not true?
> > > > >
> > > > > Jason
> > > > >
> > > > > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > >> The topics get automagically created if they don't exist.
>  Will
> > > > > > that still exist in 0.8?
> > > > > >
> > > > > > In 0.8, this is controlled by the auto.create.topics.enable
> config
> > on
> > > > the
> > > > > > brokers. If this is set to true, topics will be created when a
> > topic
> > > > > > metadata request is sent for a new topic. This feature is
> provided
> > to
> > > > aid
> > > > > > with the 0.7->0.8 migration tool and the mirror maker tools.
> These
> > > > tools
> > > > > > copy data for almost all topics from one cluster to the other, so
> > > auto
> > > > > > creating topics makes it much easier to migrate data between

 

Hi Jun, yes, we're using zk-based producers.  Producers have all been
restarted (they get cycled many times a day as well).

The specific producers that generated all those bogus topics have
definitely been restarted, etc.

Using zkCli.sh, I do see all the deleted topics, e.g.:

[zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
[<1000's of deleted topics>,<also all the remaining valid topics>]

I see also all the deleted topics under /consumers/<appname>/owners/

Jason

On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Are you using ZK-based producer? If so, those watches could be left by the
> producers if they haven't been restarted. Could you also use zkCli.sh to
> see if deleted topics are there in ZK?
>
> Thanks,
>
> Jun
>
> On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Jun,
> >
> > So, I connected to zookeeper just using telnet, and using the 4 letter
> > commands.
> >
> > If I do a dump:
> >
> > I do not see anything but valid topics, and valid consumer/owners
> mappings.
> >
> > If I check watches, I see all the 1000's of bogus topics, e.c.:
> >
> > wchc:
> > ....
> > /brokers/topics/<obsolete-topic>
> > ....
> >
> > or
> >
> > wchp
> > ....
> > /brokers/topics/<obsolete-topic>.
> > 0x383d6fe1b83f0008
> > 0x383d6fe1b83f0009
> > 0x7c3d6f9020d00041
> > 0x833d6fe2826e0000
> > 0x383d6fe1b83f000a
> > ....
> >
> > The kafka and zookeeper servers, as well as the consumer apps, are rolled
> > several times a day, automatically.
> >
> > I'm not sure how to remove the watches, is that what I need to do?
> >
> > Since this is a highly available system, can I bring down each zk server
> 1
> > at a time, and clear the bogus topics (or watches)?  How do I do this?
> >
> > Jason
> >
> >
> >
> > On Fri, Mar 15, 2013 at 8:17 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Could you check if the following path for a deleted topic exists in ZK?
> > It
> > > should have no children.
> > >
> > > /brokers/topics/[topic]
> > >
> > > If this is the case, try manually removing those paths from ZK (when
> the
> > > brokers and the consumers are down).
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Hi Neha,
> > > >
> > > > So I did this, but I still see the full list of topics (most of which
> > > have
> > > > been deleted), in the consumer logs, e.g.:
> > > >
> > > > consumer.ZookeeperConsumerConnector -
> > > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb
> Topics
> > to
> > > > consume = List(<a bazillion topics, most of which have been deleted
> > > listed
> > > > here>)
> > > >
> > > > I select topics using the white list topic selector (using a
> wild-card
> > > > pattern, etc.).
> > > >
> > > > Thoughts?
> > > >
> > > > Does zookeeper keep separate info for the consumer, that needs to be
> > > > removed?
> > > >
> > > > Jason
> > > >
> > > > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > In Kafka 0.7.2, data log directory delete and then rolling restart
> > will
> > > > > work to get rid of the topic completely from zookeeper as well as
> the
> > > > > broker.
> > > > > This is because the broker registers the topics and partitions from
> > its
> > > > > data directory into zookeeper on startup.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > > wrote:
> > > > >
> > > > > > Thanks Neha,
> > > > > >
> > > > > > So are you saying that on 0.7.2, to delete a topic I need only
> > remove
> > > > > it's
> > > > > > data log directory from each broker, and the restart the brokers?
> >  Is
> > > > it
> > > > > ok
> > > > > > if it's a rolling restart?
> > > > > >
> > > > > > For some reason I thought I also had to do something on zookeeper
> > > > servers
> > > > > > too, to remove the topics.  Is that not true?
> > >
 

Jason,

Use zkCli.sh to see if there is any child node in ZK under
/brokers/topics/deletedtopic. You should see none since you already removed
those topics from brokers' log. If so, you can manually remove those paths
in ZK using zkCli.sh (stop the consumers first). After the consumers
restart, you should not see the the deleted topics in the log of the
consumers any more.

consumer.ZookeeperConsumerConnector -
samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
consume = List(<a bazillion topics, most of which have been deleted listed
here>)

Thanks,

Jun

On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Hi Jun, yes, we're using zk-based producers.  Producers have all been
> restarted (they get cycled many times a day as well).
>
> The specific producers that generated all those bogus topics have
> definitely been restarted, etc.
>
> Using zkCli.sh, I do see all the deleted topics, e.g.:
>
> [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
> [<1000's of deleted topics>,<also all the remaining valid topics>]
>
> I see also all the deleted topics under /consumers/<appname>/owners/
>
> Jason
>
> On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Are you using ZK-based producer? If so, those watches could be left by
> the
> > producers if they haven't been restarted. Could you also use zkCli.sh to
> > see if deleted topics are there in ZK?
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Jun,
> > >
> > > So, I connected to zookeeper just using telnet, and using the 4 letter
> > > commands.
> > >
> > > If I do a dump:
> > >
> > > I do not see anything but valid topics, and valid consumer/owners
> > mappings.
> > >
> > > If I check watches, I see all the 1000's of bogus topics, e.c.:
> > >
> > > wchc:
> > > ....
> > > /brokers/topics/<obsolete-topic>
> > > ....
> > >
> > > or
> > >
> > > wchp
> > > ....
> > > /brokers/topics/<obsolete-topic>.
> > > 0x383d6fe1b83f0008
> > > 0x383d6fe1b83f0009
> > > 0x7c3d6f9020d00041
> > > 0x833d6fe2826e0000
> > > 0x383d6fe1b83f000a
> > > ....
> > >
> > > The kafka and zookeeper servers, as well as the consumer apps, are
> rolled
> > > several times a day, automatically.
> > >
> > > I'm not sure how to remove the watches, is that what I need to do?
> > >
> > > Since this is a highly available system, can I bring down each zk
> server
> > 1
> > > at a time, and clear the bogus topics (or watches)?  How do I do this?
> > >
> > > Jason
> > >
> > >
> > >
> > > On Fri, Mar 15, 2013 at 8:17 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > > > Could you check if the following path for a deleted topic exists in
> ZK?
> > > It
> > > > should have no children.
> > > >
> > > > /brokers/topics/[topic]
> > > >
> > > > If this is the case, try manually removing those paths from ZK (when
> > the
> > > > brokers and the consumers are down).
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Hi Neha,
> > > > >
> > > > > So I did this, but I still see the full list of topics (most of
> which
> > > > have
> > > > > been deleted), in the consumer logs, e.g.:
> > > > >
> > > > > consumer.ZookeeperConsumerConnector -
> > > > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb
> > Topics
> > > to
> > > > > consume = List(<a bazillion topics, most of which have been deleted
> > > > listed
> > > > > here>)
> > > > >
> > > > > I select topics using the white list topic selector (using a
> > wild-card
> > > > > pattern, etc.).
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Does zookeeper keep separate info for the consumer, that needs to
> be
> > > > > removed?
> > > > >
> > > > > Jason
> > > > >
> > > > > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <
> > > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > In Kafka 0.7.2, data log directory delete and then rolling

 

Jun,

There are indeed no nodes under /brokers/topics/deletedtopic

Also, do I need to remove the deleted apps from the
/consumers/<appname>/owners path?

So, should this be considered a bug, or not?  Essentially, each broker
knows what topics it has, but the info seems to have not propagated to zk
at all?  Is that right?

Although from the telnet interface, the removed topics don't show up under
a 'dump' command, but they do under wchc, etc.  What's that about?

Should the original procedure have been to: (1) stop all consumers, then
(2) remove the topic files on the brokers, then (3) restart the brokers,
then (4) restart the consumers?

The consumers have been up and down since the topic files were removed, but
just not ever at the same time the brokers were down....

Jason
On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Jason,
>
> Use zkCli.sh to see if there is any child node in ZK under
> /brokers/topics/deletedtopic. You should see none since you already removed
> those topics from brokers' log. If so, you can manually remove those paths
> in ZK using zkCli.sh (stop the consumers first). After the consumers
> restart, you should not see the the deleted topics in the log of the
> consumers any more.
>
> consumer.ZookeeperConsumerConnector -
> samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
> consume = List(<a bazillion topics, most of which have been deleted listed
> here>)
>
> Thanks,
>
> Jun
>
> On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
>
> > Hi Jun, yes, we're using zk-based producers.  Producers have all been
> > restarted (they get cycled many times a day as well).
> >
> > The specific producers that generated all those bogus topics have
> > definitely been restarted, etc.
> >
> > Using zkCli.sh, I do see all the deleted topics, e.g.:
> >
> > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
> > [<1000's of deleted topics>,<also all the remaining valid topics>]
> >
> > I see also all the deleted topics under /consumers/<appname>/owners/
> >
> > Jason
> >
> > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Are you using ZK-based producer? If so, those watches could be left by
> > the
> > > producers if they haven't been restarted. Could you also use zkCli.sh
> to
> > > see if deleted topics are there in ZK?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Jun,
> > > >
> > > > So, I connected to zookeeper just using telnet, and using the 4
> letter
> > > > commands.
> > > >
> > > > If I do a dump:
> > > >
> > > > I do not see anything but valid topics, and valid consumer/owners
> > > mappings.
> > > >
> > > > If I check watches, I see all the 1000's of bogus topics, e.c.:
> > > >
> > > > wchc:
> > > > ....
> > > > /brokers/topics/<obsolete-topic>
> > > > ....
> > > >
> > > > or
> > > >
> > > > wchp
> > > > ....
> > > > /brokers/topics/<obsolete-topic>.
> > > > 0x383d6fe1b83f0008
> > > > 0x383d6fe1b83f0009
> > > > 0x7c3d6f9020d00041
> > > > 0x833d6fe2826e0000
> > > > 0x383d6fe1b83f000a
> > > > ....
> > > >
> > > > The kafka and zookeeper servers, as well as the consumer apps, are
> > rolled
> > > > several times a day, automatically.
> > > >
> > > > I'm not sure how to remove the watches, is that what I need to do?
> > > >
> > > > Since this is a highly available system, can I bring down each zk
> > server
> > > 1
> > > > at a time, and clear the bogus topics (or watches)?  How do I do
> this?
> > > >
> > > > Jason
> > > >
> > > >
> > > >
> > > > On Fri, Mar 15, 2013 at 8:17 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Could you check if the following path for a deleted topic exists in
> > ZK?
> > > > It
> > > > > should have no children.
> > > > >
> > > > > /brokers/topics/[topic]
> > > > >
> > > > > If this is the case, try manually removing those paths from ZK
> (when
> > > the
> > > > > brokers and the consumers are down).

 

Jason,

This is mainly a problem that we don't have a formal way of deleting a
topic in 0.7, which we are trying to fix in 0.8.

The extra watchers on those deleted topics are likely registered by the
producers. They should be gone once /brokers/topics/deletedtopic are
removed from ZK.

You probably don't need to clean /consumers/<appname>/owners.

Thanks,

Jun
On Sun, Mar 17, 2013 at 11:11 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Jun,
>
> There are indeed no nodes under /brokers/topics/deletedtopic
>
> Also, do I need to remove the deleted apps from the
> /consumers/<appname>/owners path?
>
> So, should this be considered a bug, or not?  Essentially, each broker
> knows what topics it has, but the info seems to have not propagated to zk
> at all?  Is that right?
>
> Although from the telnet interface, the removed topics don't show up under
> a 'dump' command, but they do under wchc, etc.  What's that about?
>
> Should the original procedure have been to: (1) stop all consumers, then
> (2) remove the topic files on the brokers, then (3) restart the brokers,
> then (4) restart the consumers?
>
> The consumers have been up and down since the topic files were removed, but
> just not ever at the same time the brokers were down....
>
> Jason
>
>
> On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Jason,
> >
> > Use zkCli.sh to see if there is any child node in ZK under
> > /brokers/topics/deletedtopic. You should see none since you already
> removed
> > those topics from brokers' log. If so, you can manually remove those
> paths
> > in ZK using zkCli.sh (stop the consumers first). After the consumers
> > restart, you should not see the the deleted topics in the log of the
> > consumers any more.
> >
> > consumer.ZookeeperConsumerConnector -
> > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to
> > consume = List(<a bazillion topics, most of which have been deleted
> listed
> > here>)
> >
> > Thanks,
> >
> > Jun
> >
> > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi Jun, yes, we're using zk-based producers.  Producers have all been
> > > restarted (they get cycled many times a day as well).
> > >
> > > The specific producers that generated all those bogus topics have
> > > definitely been restarted, etc.
> > >
> > > Using zkCli.sh, I do see all the deleted topics, e.g.:
> > >
> > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
> > > [<1000's of deleted topics>,<also all the remaining valid topics>]
> > >
> > > I see also all the deleted topics under /consumers/<appname>/owners/
> > >
> > > Jason
> > >
> > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > > > Are you using ZK-based producer? If so, those watches could be left
> by
> > > the
> > > > producers if they haven't been restarted. Could you also use zkCli.sh
> > to
> > > > see if deleted topics are there in ZK?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > So, I connected to zookeeper just using telnet, and using the 4
> > letter
> > > > > commands.
> > > > >
> > > > > If I do a dump:
> > > > >
> > > > > I do not see anything but valid topics, and valid consumer/owners
> > > > mappings.
> > > > >
> > > > > If I check watches, I see all the 1000's of bogus topics, e.c.:
> > > > >
> > > > > wchc:
> > > > > ....
> > > > > /brokers/topics/<obsolete-topic>
> > > > > ....
> > > > >
> > > > > or
> > > > >
> > > > > wchp
> > > > > ....
> > > > > /brokers/topics/<obsolete-topic>.
> > > > > 0x383d6fe1b83f0008
> > > > > 0x383d6fe1b83f0009
> > > > > 0x7c3d6f9020d00041
> > > > > 0x833d6fe2826e0000
> > > > > 0x383d6fe1b83f000a
> > > > > ....
> > > > >
> > > > > The kafka and zookeeper servers, as well as the consumer apps, are
> > > rolled
> > > > > several times a day, automatically.
> > > > >
> > > > > I'm not sure how to remove the watches, is that what I need to do?

 

Thanks Jun,

I have nothing listed under /brokers/topics/deletedtopic, etc....so that
doesn't appear to be the issue.

I will try removing now the unwanted topics under /brokers/topics/.

Jason
On Mon, Mar 18, 2013 at 9:31 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Jason,
>
> This is mainly a problem that we don't have a formal way of deleting a
> topic in 0.7, which we are trying to fix in 0.8.
>
> The extra watchers on those deleted topics are likely registered by the
> producers. They should be gone once /brokers/topics/deletedtopic are
> removed from ZK.
>
> You probably don't need to clean /consumers/<appname>/owners.
>
> Thanks,
>
> Jun
>
>
> On Sun, Mar 17, 2013 at 11:11 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
>
> > Jun,
> >
> > There are indeed no nodes under /brokers/topics/deletedtopic
> >
> > Also, do I need to remove the deleted apps from the
> > /consumers/<appname>/owners path?
> >
> > So, should this be considered a bug, or not?  Essentially, each broker
> > knows what topics it has, but the info seems to have not propagated to zk
> > at all?  Is that right?
> >
> > Although from the telnet interface, the removed topics don't show up
> under
> > a 'dump' command, but they do under wchc, etc.  What's that about?
> >
> > Should the original procedure have been to: (1) stop all consumers, then
> > (2) remove the topic files on the brokers, then (3) restart the brokers,
> > then (4) restart the consumers?
> >
> > The consumers have been up and down since the topic files were removed,
> but
> > just not ever at the same time the brokers were down....
> >
> > Jason
> >
> >
> > On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Jason,
> > >
> > > Use zkCli.sh to see if there is any child node in ZK under
> > > /brokers/topics/deletedtopic. You should see none since you already
> > removed
> > > those topics from brokers' log. If so, you can manually remove those
> > paths
> > > in ZK using zkCli.sh (stop the consumers first). After the consumers
> > > restart, you should not see the the deleted topics in the log of the
> > > consumers any more.
> > >
> > > consumer.ZookeeperConsumerConnector -
> > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics
> to
> > > consume = List(<a bazillion topics, most of which have been deleted
> > listed
> > > here>)
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Hi Jun, yes, we're using zk-based producers.  Producers have all been
> > > > restarted (they get cycled many times a day as well).
> > > >
> > > > The specific producers that generated all those bogus topics have
> > > > definitely been restarted, etc.
> > > >
> > > > Using zkCli.sh, I do see all the deleted topics, e.g.:
> > > >
> > > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
> > > > [<1000's of deleted topics>,<also all the remaining valid topics>]
> > > >
> > > > I see also all the deleted topics under /consumers/<appname>/owners/
> > > >
> > > > Jason
> > > >
> > > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Are you using ZK-based producer? If so, those watches could be left
> > by
> > > > the
> > > > > producers if they haven't been restarted. Could you also use
> zkCli.sh
> > > to
> > > > > see if deleted topics are there in ZK?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > So, I connected to zookeeper just using telnet, and using the 4
> > > letter
> > > > > > commands.
> > > > > >
> > > > > > If I do a dump:
> > > > > >
> > > > > > I do not see anything but valid topics, and valid consumer/owners
> > > > > mappings.
> > > > > >
> > > > > > If I check watches, I see all the 1000's of bogus topics, e.c.:
> > > > > >
> > > > > > wchc:
> > > > > > ....
> >
 

In 0.8, the delete topic command will be able to remove a topic in an
online cluster, without having to shutdown anything.

Thanks,
Neha
On Mon, Mar 18, 2013 at 1:37 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Thanks Jun,
>
> I have nothing listed under /brokers/topics/deletedtopic, etc....so that
> doesn't appear to be the issue.
>
> I will try removing now the unwanted topics under /brokers/topics/.
>
> Jason
>
>
> On Mon, Mar 18, 2013 at 9:31 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Jason,
> >
> > This is mainly a problem that we don't have a formal way of deleting a
> > topic in 0.7, which we are trying to fix in 0.8.
> >
> > The extra watchers on those deleted topics are likely registered by the
> > producers. They should be gone once /brokers/topics/deletedtopic are
> > removed from ZK.
> >
> > You probably don't need to clean /consumers/<appname>/owners.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Sun, Mar 17, 2013 at 11:11 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Jun,
> > >
> > > There are indeed no nodes under /brokers/topics/deletedtopic
> > >
> > > Also, do I need to remove the deleted apps from the
> > > /consumers/<appname>/owners path?
> > >
> > > So, should this be considered a bug, or not?  Essentially, each broker
> > > knows what topics it has, but the info seems to have not propagated to
> zk
> > > at all?  Is that right?
> > >
> > > Although from the telnet interface, the removed topics don't show up
> > under
> > > a 'dump' command, but they do under wchc, etc.  What's that about?
> > >
> > > Should the original procedure have been to: (1) stop all consumers,
> then
> > > (2) remove the topic files on the brokers, then (3) restart the
> brokers,
> > > then (4) restart the consumers?
> > >
> > > The consumers have been up and down since the topic files were removed,
> > but
> > > just not ever at the same time the brokers were down....
> > >
> > > Jason
> > >
> > >
> > > On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > > > Jason,
> > > >
> > > > Use zkCli.sh to see if there is any child node in ZK under
> > > > /brokers/topics/deletedtopic. You should see none since you already
> > > removed
> > > > those topics from brokers' log. If so, you can manually remove those
> > > paths
> > > > in ZK using zkCli.sh (stop the consumers first). After the consumers
> > > > restart, you should not see the the deleted topics in the log of the
> > > > consumers any more.
> > > >
> > > > consumer.ZookeeperConsumerConnector -
> > > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb
> Topics
> > to
> > > > consume = List(<a bazillion topics, most of which have been deleted
> > > listed
> > > > here>)
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > > wrote:
> > > >
> > > > > Hi Jun, yes, we're using zk-based producers.  Producers have all
> been
> > > > > restarted (they get cycled many times a day as well).
> > > > >
> > > > > The specific producers that generated all those bogus topics have
> > > > > definitely been restarted, etc.
> > > > >
> > > > > Using zkCli.sh, I do see all the deleted topics, e.g.:
> > > > >
> > > > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
> > > > > [<1000's of deleted topics>,<also all the remaining valid topics>]
> > > > >
> > > > > I see also all the deleted topics under
> /consumers/<appname>/owners/
> > > > >
> > > > > Jason
> > > > >
> > > > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > Are you using ZK-based producer? If so, those watches could be
> left
> > > by
> > > > > the
> > > > > > producers if they haven't been restarted. Could you also use
> > zkCli.sh
> > > > to
> > > > > > see if deleted topics are there in ZK?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > > On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <
> [EMAIL PROTECTED]
> >
 

One other clue, is that although those topics still show up under
/brokers/topics/<topicname>/ they contain empty sub-nodes.....So zk knows
they don't exist on any broker.  So maybe that's the issue.  Brokers are
successfully removing themselves as serving a topic, but the topic itself
remains.  The consumers then just see the topics (even with no brokers
listed) and continue to watch the topic.

Jason

On Mon, Mar 18, 2013 at 1:37 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Thanks Jun,
>
> I have nothing listed under /brokers/topics/deletedtopic, etc....so that
> doesn't appear to be the issue.
>
> I will try removing now the unwanted topics under /brokers/topics/.
>
> Jason
>
>
> On Mon, Mar 18, 2013 at 9:31 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
>> Jason,
>>
>> This is mainly a problem that we don't have a formal way of deleting a
>> topic in 0.7, which we are trying to fix in 0.8.
>>
>> The extra watchers on those deleted topics are likely registered by the
>> producers. They should be gone once /brokers/topics/deletedtopic are
>> removed from ZK.
>>
>> You probably don't need to clean /consumers/<appname>/owners.
>>
>> Thanks,
>>
>> Jun
>>
>>
>> On Sun, Mar 17, 2013 at 11:11 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> wrote:
>>
>> > Jun,
>> >
>> > There are indeed no nodes under /brokers/topics/deletedtopic
>> >
>> > Also, do I need to remove the deleted apps from the
>> > /consumers/<appname>/owners path?
>> >
>> > So, should this be considered a bug, or not?  Essentially, each broker
>> > knows what topics it has, but the info seems to have not propagated to
>> zk
>> > at all?  Is that right?
>> >
>> > Although from the telnet interface, the removed topics don't show up
>> under
>> > a 'dump' command, but they do under wchc, etc.  What's that about?
>> >
>> > Should the original procedure have been to: (1) stop all consumers, then
>> > (2) remove the topic files on the brokers, then (3) restart the brokers,
>> > then (4) restart the consumers?
>> >
>> > The consumers have been up and down since the topic files were removed,
>> but
>> > just not ever at the same time the brokers were down....
>> >
>> > Jason
>> >
>> >
>> > On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> >
>> > > Jason,
>> > >
>> > > Use zkCli.sh to see if there is any child node in ZK under
>> > > /brokers/topics/deletedtopic. You should see none since you already
>> > removed
>> > > those topics from brokers' log. If so, you can manually remove those
>> > paths
>> > > in ZK using zkCli.sh (stop the consumers first). After the consumers
>> > > restart, you should not see the the deleted topics in the log of the
>> > > consumers any more.
>> > >
>> > > consumer.ZookeeperConsumerConnector -
>> > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb
>> Topics to
>> > > consume = List(<a bazillion topics, most of which have been deleted
>> > listed
>> > > here>)
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> > > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]>
>> > > wrote:
>> > >
>> > > > Hi Jun, yes, we're using zk-based producers.  Producers have all
>> been
>> > > > restarted (they get cycled many times a day as well).
>> > > >
>> > > > The specific producers that generated all those bogus topics have
>> > > > definitely been restarted, etc.
>> > > >
>> > > > Using zkCli.sh, I do see all the deleted topics, e.g.:
>> > > >
>> > > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
>> > > > [<1000's of deleted topics>,<also all the remaining valid topics>]
>> > > >
>> > > > I see also all the deleted topics under /consumers/<appname>/owners/
>> > > >
>> > > > Jason
>> > > >
>> > > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>> > > >
>> > > > > Are you using ZK-based producer? If so, those watches could be
>> left
>> > by
>> > > > the
>> > > > > producers if they haven't been restarted. Could you also use
>> zkCli.sh
>> > > to
>> > > > > see if deleted topics are there in ZK?

 

So, looks like in 0.7, the way to correctly delete topics from the brokers
and zookeeper is as follows -

1. Shutdown producers, consumers and brokers.
2. Delete the topic logs from the brokers
3. Delete the /brokers/topics/<topic> nodes from zookeeper
4. Restart the brokers
5. Restart producers and consumers.

Thanks,
Neha
On Mon, Mar 18, 2013 at 9:31 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Jason,
>
> This is mainly a problem that we don't have a formal way of deleting a
> topic in 0.7, which we are trying to fix in 0.8.
>
> The extra watchers on those deleted topics are likely registered by the
> producers. They should be gone once /brokers/topics/deletedtopic are
> removed from ZK.
>
> You probably don't need to clean /consumers/<appname>/owners.
>
> Thanks,
>
> Jun
>
>
> On Sun, Mar 17, 2013 at 11:11 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
>
> > Jun,
> >
> > There are indeed no nodes under /brokers/topics/deletedtopic
> >
> > Also, do I need to remove the deleted apps from the
> > /consumers/<appname>/owners path?
> >
> > So, should this be considered a bug, or not?  Essentially, each broker
> > knows what topics it has, but the info seems to have not propagated to zk
> > at all?  Is that right?
> >
> > Although from the telnet interface, the removed topics don't show up
> under
> > a 'dump' command, but they do under wchc, etc.  What's that about?
> >
> > Should the original procedure have been to: (1) stop all consumers, then
> > (2) remove the topic files on the brokers, then (3) restart the brokers,
> > then (4) restart the consumers?
> >
> > The consumers have been up and down since the topic files were removed,
> but
> > just not ever at the same time the brokers were down....
> >
> > Jason
> >
> >
> > On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> >
> > > Jason,
> > >
> > > Use zkCli.sh to see if there is any child node in ZK under
> > > /brokers/topics/deletedtopic. You should see none since you already
> > removed
> > > those topics from brokers' log. If so, you can manually remove those
> > paths
> > > in ZK using zkCli.sh (stop the consumers first). After the consumers
> > > restart, you should not see the the deleted topics in the log of the
> > > consumers any more.
> > >
> > > consumer.ZookeeperConsumerConnector -
> > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics
> to
> > > consume = List(<a bazillion topics, most of which have been deleted
> > listed
> > > here>)
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > >
> > > > Hi Jun, yes, we're using zk-based producers.  Producers have all been
> > > > restarted (they get cycled many times a day as well).
> > > >
> > > > The specific producers that generated all those bogus topics have
> > > > definitely been restarted, etc.
> > > >
> > > > Using zkCli.sh, I do see all the deleted topics, e.g.:
> > > >
> > > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
> > > > [<1000's of deleted topics>,<also all the remaining valid topics>]
> > > >
> > > > I see also all the deleted topics under /consumers/<appname>/owners/
> > > >
> > > > Jason
> > > >
> > > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > >
> > > > > Are you using ZK-based producer? If so, those watches could be left
> > by
> > > > the
> > > > > producers if they haven't been restarted. Could you also use
> zkCli.sh
> > > to
> > > > > see if deleted topics are there in ZK?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <[EMAIL PROTECTED]
> >
> > > > wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > So, I connected to zookeeper just using telnet, and using the 4
> > > letter
> > > > > > commands.
> > > > > >
> > > > > > If I do a dump:
> > > > > >
> > > > > > I do not see anything but valid topics, and valid consumer/owners
> > > > > mappings.
> > > > > >

 

Ok,

So we'll leave it as this (I won't file a bug).  It is of course less than
ideal to require everything to be brought down when deleting topics!  Since
we consider the cluster to be a high-availability resource.  I assume in
0.8, it won't be necessary to bring everything down to delete a topic?

Jason

On Mon, Mar 18, 2013 at 11:05 AM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> So, looks like in 0.7, the way to correctly delete topics from the brokers
> and zookeeper is as follows -
>
> 1. Shutdown producers, consumers and brokers.
> 2. Delete the topic logs from the brokers
> 3. Delete the /brokers/topics/<topic> nodes from zookeeper
> 4. Restart the brokers
> 5. Restart producers and consumers.
>
> Thanks,
> Neha
>
>
> On Mon, Mar 18, 2013 at 9:31 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > Jason,
> >
> > This is mainly a problem that we don't have a formal way of deleting a
> > topic in 0.7, which we are trying to fix in 0.8.
> >
> > The extra watchers on those deleted topics are likely registered by the
> > producers. They should be gone once /brokers/topics/deletedtopic are
> > removed from ZK.
> >
> > You probably don't need to clean /consumers/<appname>/owners.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Sun, Mar 17, 2013 at 11:11 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Jun,
> > >
> > > There are indeed no nodes under /brokers/topics/deletedtopic
> > >
> > > Also, do I need to remove the deleted apps from the
> > > /consumers/<appname>/owners path?
> > >
> > > So, should this be considered a bug, or not?  Essentially, each broker
> > > knows what topics it has, but the info seems to have not propagated to
> zk
> > > at all?  Is that right?
> > >
> > > Although from the telnet interface, the removed topics don't show up
> > under
> > > a 'dump' command, but they do under wchc, etc.  What's that about?
> > >
> > > Should the original procedure have been to: (1) stop all consumers,
> then
> > > (2) remove the topic files on the brokers, then (3) restart the
> brokers,
> > > then (4) restart the consumers?
> > >
> > > The consumers have been up and down since the topic files were removed,
> > but
> > > just not ever at the same time the brokers were down....
> > >
> > > Jason
> > >
> > >
> > > On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > >
> > > > Jason,
> > > >
> > > > Use zkCli.sh to see if there is any child node in ZK under
> > > > /brokers/topics/deletedtopic. You should see none since you already
> > > removed
> > > > those topics from brokers' log. If so, you can manually remove those
> > > paths
> > > > in ZK using zkCli.sh (stop the consumers first). After the consumers
> > > > restart, you should not see the the deleted topics in the log of the
> > > > consumers any more.
> > > >
> > > > consumer.ZookeeperConsumerConnector -
> > > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb
> Topics
> > to
> > > > consume = List(<a bazillion topics, most of which have been deleted
> > > listed
> > > > here>)
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > > wrote:
> > > >
> > > > > Hi Jun, yes, we're using zk-based producers.  Producers have all
> been
> > > > > restarted (they get cycled many times a day as well).
> > > > >
> > > > > The specific producers that generated all those bogus topics have
> > > > > definitely been restarted, etc.
> > > > >
> > > > > Using zkCli.sh, I do see all the deleted topics, e.g.:
> > > > >
> > > > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics
> > > > > [<1000's of deleted topics>,<also all the remaining valid topics>]
> > > > >
> > > > > I see also all the deleted topics under
> /consumers/<appname>/owners/
> > > > >
> > > > > Jason
> > > > >
> > > > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > > > Are you using ZK-based producer? If so, those watches could be
> left
> > > by
> > > > > the

 

On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> In 0.8, this is controlled by the auto.create.topics.enable config on the
> brokers. If this is set to true, topics will be created when a topic
> metadata request is sent for a new topic. This feature is provided to aid
> with the 0.7->0.8 migration tool and the mirror maker tools. These tools
> copy data for almost all topics from one cluster to the other, so auto
> creating topics makes it much easier to migrate data between clusters.
I just want to add, that we have a use case where it is nice to able to
have auto-created topics, as part of the design in regular operation.
 Thus, I'd hope the 'auto.create.topics.enable' has a purpose beyond just
migrating data, etc!  It should be a first class use case, in my view.

Thanks!