Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> auto-delete empty, old topics?


Copy link to this message
-
Re: auto-delete empty, old topics?
Yes, you're description matches what I did.  And the brokers have been
bounced many times since then (they are auto-deployed many times a day,
etc.).  And the consumers also have been restarted many times since then.

Could it be related to using the white-list topic selector, etc.?

Let me know if you need me to look a the state of my zookeeper data....

Jason

On Thu, Mar 14, 2013 at 2:39 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Jason,
>
> Let me see if I understood what you did here. In Kafka 0.7.2, you deleted
> the Kafka log files from the server and bounced the broker. This should've
> ideally deleted those topics from zookeeper, the consumer reads the same
> zookeeper paths that the broker writes. Doing this should cause rebalancing
> on the consumer and it should readjust its consumption to include the
> remaining topics. But looks like that didn't happen somehow ?
>
> I agree that log line is redundant, I think it is removed in 0.8
>
> Thanks,
> Neha
>
>
> On Thu, Mar 14, 2013 at 2:17 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:
>
> > Also,
> >
> > I see a bazillion consecutive log lines like this:
> >
> > 2013-03-14 19:54:13,306  INFO [Thread-4] consumer.ConsumerIterator -
> > Clearing the current data chunk for this consumer iterator
> >
> > With the same content (not sure how useful that is!).
> >
> > Jason
> >
> > On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Hi Neha,
> > >
> > > So I did this, but I still see the full list of topics (most of which
> > have
> > > been deleted), in the consumer logs, e.g.:
> > >
> > > consumer.ZookeeperConsumerConnector -
> > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics
> to
> > > consume = List(<a bazillion topics, most of which have been deleted
> > listed
> > > here>)
> > >
> > > I select topics using the white list topic selector (using a wild-card
> > > pattern, etc.).
> > >
> > > Thoughts?
> > >
> > > Does zookeeper keep separate info for the consumer, that needs to be
> > > removed?
> > >
> > > Jason
> > >
> > > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > >wrote:
> > >
> > >> In Kafka 0.7.2, data log directory delete and then rolling restart
> will
> > >> work to get rid of the topic completely from zookeeper as well as the
> > >> broker.
> > >> This is because the broker registers the topics and partitions from
> its
> > >> data directory into zookeeper on startup.
> > >>
> > >> Thanks,
> > >> Neha
> > >>
> > >>
> > >> On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >>
> > >> > Thanks Neha,
> > >> >
> > >> > So are you saying that on 0.7.2, to delete a topic I need only
> remove
> > >> it's
> > >> > data log directory from each broker, and the restart the brokers?
>  Is
> > >> it ok
> > >> > if it's a rolling restart?
> > >> >
> > >> > For some reason I thought I also had to do something on zookeeper
> > >> servers
> > >> > too, to remove the topics.  Is that not true?
> > >> >
> > >> > Jason
> > >> >
> > >> > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > >> > >wrote:
> > >> >
> > >> > > >> The topics get automagically created if they don't exist.  Will
> > >> > > that still exist in 0.8?
> > >> > >
> > >> > > In 0.8, this is controlled by the auto.create.topics.enable config
> > on
> > >> the
> > >> > > brokers. If this is set to true, topics will be created when a
> topic
> > >> > > metadata request is sent for a new topic. This feature is provided
> > to
> > >> aid
> > >> > > with the 0.7->0.8 migration tool and the mirror maker tools. These
> > >> tools
> > >> > > copy data for almost all topics from one cluster to the other, so
> > auto
> > >> > > creating topics makes it much easier to migrate data between
> > clusters.
> > >> > >
> > >> > > >> What's the procedure for deleting topics in kafka 0.7.2?
> > >> > >
> > >> > > There is no official way to delete a topic. You have to delete the