Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Is there a way to pull out kafka metadata from zookeeper?


Copy link to this message
-
Re: Is there a way to pull out kafka metadata from zookeeper?
>> For each consumer consumes different
topic/replica I have to specify those 20 brokers and go over all of them to
know which broker is alive. And even worse how about I dynamically add new
broker into the cluster and remove the old one

TopicMetadataRequest is a batch API and you can get metadata information
for either a list of all topics or all topics in the cluster, if you
specify an empty list of topics. Adding a broker is not a problem since the
metadata request also returns the list of brokers in a cluster. The reason
this is better than reading from zookeeper is because the same operation
would require multiple zookeeper roundtrips, instead of a single
TopicMetadataRequest roundtrip to some kafka broker.

Thanks,
Neha
On Fri, Oct 11, 2013 at 11:30 AM, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:

> Thanks guys!
> But I feel weird. Assume I have 20 brokers for 10 different topics with 2
> partitions and  2 replicas for each. For each consumer consumes different
> topic/replica I have to specify those 20 brokers and go over all of them to
> know which broker is alive. And even worse how about I dynamically add new
> broker into the cluster and remove the old one. I think it's nice to have a
> way to get metadata from zookeeper(centralized coordinator?) directly.
>
> Best,
> Siyuan
>
>
> On Fri, Oct 11, 2013 at 9:12 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > If, for some reason, you don't have access to a virtual IP or load
> > balancer, you need to round robin once through all the brokers before
> > failing a TopicMetadataRequest. So unless all the brokers in your cluster
> > are down, this should not be a problem.
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Oct 10, 2013 at 10:50 PM, [EMAIL PROTECTED] <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi guys,
> > >
> > > I'm trying to maintain a bunch of simple kafka consumer to consume
> > messages
> > > from brokers. I know there is a way to send TopicMetadataRequest to
> > broker
> > > and get the response from the broker. But you have to specify the
> broker
> > > list to query the information. But broker might not be available
> because
> > of
> > > some failure. My question is is there any api I can call and query
> broker
> > > metadata for topic/partition directly from zookeeper? I know I can
> query
> > > that information using zookeeper API. But that's not friendly
> > datastructure
> > >  like the TopicMetadata/PartitionMetadata.  Thank you!
> > >
> > > Best,
> > > Siyuan
> > >
> >
>

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB