Kafka, mail # user - Re: How to get topic list from kafka server - 2013-03-13, 12:10
 Search Hadoop and all its subprojects:

Switch to Plain View
+
Snehalata Nagaje 2013-03-13, 07:39
Copy link to this message
-
Re: How to get topic list from kafka server
I'm curious the use case, but I think you'll need to use a combination of
ZooKeeper meta data and the TopicMetadata API.

The list of known topics for the cluster is in ZooKeeper:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper

Also in ZooKeeper is the list of Brokers.

To protect yourself from changes on how Kafka uses ZooKeeper I'd do the
following:
- get the list of topics from ZooKeeper
- get the list of Brokers from ZooKeeper
- connect to the first Broker and use the list of topics to send
TopicMetadataRequest requests to figure out which partitions for the topic
are on which Broker (leader is what you want to know I think)

Since the leader assignments may change, be prepared for 'where' a
topic/partition lives to change. For example the Broker you are talking too
may suddenly be the leader for another topic/partition if that
topic/partition's leader failed and the current Broker was elected leader.
Depending on why you need to know about every topic/partition on a Broker
may have to listen on ZooKeeper events to see that this transition happened.

Hope this helps,
Chris
On Wed, Mar 13, 2013 at 3:38 AM, Snehalata Nagaje <
[EMAIL PROTECTED]> wrote:
 
+
Jun Rao 2013-03-13, 14:36
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB