Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> auto-delete empty, old topics?


Copy link to this message
-
Re: auto-delete empty, old topics?
This could be a bug with the topic discovery logic in the wildcard
consumer. Please can you file a bug and attach your consumer logs there ?

Thanks,
Neha
On Thu, Mar 14, 2013 at 3:20 PM, Jason Rosenberg <[EMAIL PROTECTED]> wrote:

> Yes, you're description matches what I did.  And the brokers have been
> bounced many times since then (they are auto-deployed many times a day,
> etc.).  And the consumers also have been restarted many times since then.
>
> Could it be related to using the white-list topic selector, etc.?
>
> Let me know if you need me to look a the state of my zookeeper data....
>
> Jason
>
> On Thu, Mar 14, 2013 at 2:39 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > Jason,
> >
> > Let me see if I understood what you did here. In Kafka 0.7.2, you deleted
> > the Kafka log files from the server and bounced the broker. This
> should've
> > ideally deleted those topics from zookeeper, the consumer reads the same
> > zookeeper paths that the broker writes. Doing this should cause
> rebalancing
> > on the consumer and it should readjust its consumption to include the
> > remaining topics. But looks like that didn't happen somehow ?
> >
> > I agree that log line is redundant, I think it is removed in 0.8
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Mar 14, 2013 at 2:17 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> wrote:
> >
> > > Also,
> > >
> > > I see a bazillion consecutive log lines like this:
> > >
> > > 2013-03-14 19:54:13,306  INFO [Thread-4] consumer.ConsumerIterator -
> > > Clearing the current data chunk for this consumer iterator
> > >
> > > With the same content (not sure how useful that is!).
> > >
> > > Jason
> > >
> > > On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > wrote:
> > >
> > > > Hi Neha,
> > > >
> > > > So I did this, but I still see the full list of topics (most of which
> > > have
> > > > been deleted), in the consumer logs, e.g.:
> > > >
> > > > consumer.ZookeeperConsumerConnector -
> > > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb
> Topics
> > to
> > > > consume = List(<a bazillion topics, most of which have been deleted
> > > listed
> > > > here>)
> > > >
> > > > I select topics using the white list topic selector (using a
> wild-card
> > > > pattern, etc.).
> > > >
> > > > Thoughts?
> > > >
> > > > Does zookeeper keep separate info for the consumer, that needs to be
> > > > removed?
> > > >
> > > > Jason
> > > >
> > > > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > >wrote:
> > > >
> > > >> In Kafka 0.7.2, data log directory delete and then rolling restart
> > will
> > > >> work to get rid of the topic completely from zookeeper as well as
> the
> > > >> broker.
> > > >> This is because the broker registers the topics and partitions from
> > its
> > > >> data directory into zookeeper on startup.
> > > >>
> > > >> Thanks,
> > > >> Neha
> > > >>
> > > >>
> > > >> On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg <[EMAIL PROTECTED]>
> > > wrote:
> > > >>
> > > >> > Thanks Neha,
> > > >> >
> > > >> > So are you saying that on 0.7.2, to delete a topic I need only
> > remove
> > > >> it's
> > > >> > data log directory from each broker, and the restart the brokers?
> >  Is
> > > >> it ok
> > > >> > if it's a rolling restart?
> > > >> >
> > > >> > For some reason I thought I also had to do something on zookeeper
> > > >> servers
> > > >> > too, to remove the topics.  Is that not true?
> > > >> >
> > > >> > Jason
> > > >> >
> > > >> > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede <
> > > [EMAIL PROTECTED]
> > > >> > >wrote:
> > > >> >
> > > >> > > >> The topics get automagically created if they don't exist.
>  Will
> > > >> > > that still exist in 0.8?
> > > >> > >
> > > >> > > In 0.8, this is controlled by the auto.create.topics.enable
> config
> > > on
> > > >> the
> > > >> > > brokers. If this is set to true, topics will be created when a
> > topic
> > > >> > > metadata request is sent for a new topic. This feature is

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB