Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - Is there a way to pull out kafka metadata from zookeeper?


+
hsy541@...> 2013-10-11, 05:50
+
Jun Rao 2013-10-11, 15:19
+
Neha Narkhede 2013-10-11, 16:13
+
hsy541@...> 2013-10-11, 18:31
Copy link to this message
-
Re: Is there a way to pull out kafka metadata from zookeeper?
Neha Narkhede 2013-10-11, 21:39
>> For each consumer consumes different
topic/replica I have to specify those 20 brokers and go over all of them to
know which broker is alive. And even worse how about I dynamically add new
broker into the cluster and remove the old one

TopicMetadataRequest is a batch API and you can get metadata information
for either a list of all topics or all topics in the cluster, if you
specify an empty list of topics. Adding a broker is not a problem since the
metadata request also returns the list of brokers in a cluster. The reason
this is better than reading from zookeeper is because the same operation
would require multiple zookeeper roundtrips, instead of a single
TopicMetadataRequest roundtrip to some kafka broker.

Thanks,
Neha
On Fri, Oct 11, 2013 at 11:30 AM, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:

> Thanks guys!
> But I feel weird. Assume I have 20 brokers for 10 different topics with 2
> partitions and  2 replicas for each. For each consumer consumes different
> topic/replica I have to specify those 20 brokers and go over all of them to
> know which broker is alive. And even worse how about I dynamically add new
> broker into the cluster and remove the old one. I think it's nice to have a
> way to get metadata from zookeeper(centralized coordinator?) directly.
>
> Best,
> Siyuan
>
>
> On Fri, Oct 11, 2013 at 9:12 AM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > If, for some reason, you don't have access to a virtual IP or load
> > balancer, you need to round robin once through all the brokers before
> > failing a TopicMetadataRequest. So unless all the brokers in your cluster
> > are down, this should not be a problem.
> >
> > Thanks,
> > Neha
> >
> >
> > On Thu, Oct 10, 2013 at 10:50 PM, [EMAIL PROTECTED] <[EMAIL PROTECTED]>
> > wrote:
> >
> > > Hi guys,
> > >
> > > I'm trying to maintain a bunch of simple kafka consumer to consume
> > messages
> > > from brokers. I know there is a way to send TopicMetadataRequest to
> > broker
> > > and get the response from the broker. But you have to specify the
> broker
> > > list to query the information. But broker might not be available
> because
> > of
> > > some failure. My question is is there any api I can call and query
> broker
> > > metadata for topic/partition directly from zookeeper? I know I can
> query
> > > that information using zookeeper API. But that's not friendly
> > datastructure
> > >  like the TopicMetadata/PartitionMetadata.  Thank you!
> > >
> > > Best,
> > > Siyuan
> > >
> >
>

 
+
hsy541@...> 2013-10-11, 22:20
+
Bruno D. Rodrigues 2013-10-11, 23:00
+
hsy541@...> 2013-10-12, 00:16
+
Guozhang Wang 2013-10-12, 00:50
+
Bruno D. Rodrigues 2013-10-12, 12:18
+
Neha Narkhede 2013-10-12, 18:02