Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Is there a way to pull out kafka metadata from zookeeper?


Copy link to this message
-
Re: Is there a way to pull out kafka metadata from zookeeper?
Guozhang Wang 2013-10-12, 00:50
Hello Siyuan,

Pre 0.8 Kafka does have a zookeeper based producer api, and this has been
removed in 0.8. You can find some of the reasons in this JIRA:

https://issues.apache.org/jira/browse/KAFKA-369

Guozhang
On Fri, Oct 11, 2013 at 5:16 PM, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:

> That's why I'm asking, I would like to see a kafka zookeeper client api to
> get TopicMetadata instead of my own hacky way to query the zookeeper
>
> Thanks!
> Best,
> Siyuan
>
>
> On Fri, Oct 11, 2013 at 4:00 PM, Bruno D. Rodrigues <
> [EMAIL PROTECTED]> wrote:
>
> > Why not ask zookeeper for the list of brokers and then ask a random
> > broker for the metadata (and repeat if the broker is down), even if
> > it's two calls.
> >
> > Heck it already does unnecessary connections. It connects to a broker,
> > gets the metadata, disconnects, and then connects again for the data.
> > If it's already assumed a producer or consumer will take some seconds
> > until ready, what is another call gonna prejudice the flow.
> >
> > Then producers and consumers would then be consistently configured. Or
> > allow the producers to also go to a broker instead of zookeeper.
> >
> > This way the consumer needs to know and hardcode at least one node.
> > The node can fail. It can be changed.
> >
> > I thought zookeeper served to abstract this kind of complexity
> >
> >
> >
> >
> >
> >
> >
> > --
> > Bruno Rodrigues
> > Sent from my iPhone
> >
> > No dia 11/10/2013, às 22:40, Neha Narkhede <[EMAIL PROTECTED]>
> > escreveu:
> >
> > >>> For each consumer consumes different
> > > topic/replica I have to specify those 20 brokers and go over all of
> them
> > to
> > > know which broker is alive. And even worse how about I dynamically add
> > new
> > > broker into the cluster and remove the old one
> > >
> > > TopicMetadataRequest is a batch API and you can get metadata
> information
> > > for either a list of all topics or all topics in the cluster, if you
> > > specify an empty list of topics. Adding a broker is not a problem since
> > the
> > > metadata request also returns the list of brokers in a cluster. The
> > reason
> > > this is better than reading from zookeeper is because the same
> operation
> > > would require multiple zookeeper roundtrips, instead of a single
> > > TopicMetadataRequest roundtrip to some kafka broker.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > >> On Fri, Oct 11, 2013 at 11:30 AM, [EMAIL PROTECTED] <[EMAIL PROTECTED]>
> > wrote:
> > >>
> > >> Thanks guys!
> > >> But I feel weird. Assume I have 20 brokers for 10 different topics
> with
> > 2
> > >> partitions and  2 replicas for each. For each consumer consumes
> > different
> > >> topic/replica I have to specify those 20 brokers and go over all of
> > them to
> > >> know which broker is alive. And even worse how about I dynamically add
> > new
> > >> broker into the cluster and remove the old one. I think it's nice to
> > have a
> > >> way to get metadata from zookeeper(centralized coordinator?) directly.
> > >>
> > >> Best,
> > >> Siyuan
> > >>
> > >>
> > >> On Fri, Oct 11, 2013 at 9:12 AM, Neha Narkhede <
> [EMAIL PROTECTED]
> > >>> wrote:
> > >>
> > >>> If, for some reason, you don't have access to a virtual IP or load
> > >>> balancer, you need to round robin once through all the brokers before
> > >>> failing a TopicMetadataRequest. So unless all the brokers in your
> > cluster
> > >>> are down, this should not be a problem.
> > >>>
> > >>> Thanks,
> > >>> Neha
> > >>>
> > >>>
> > >>> On Thu, Oct 10, 2013 at 10:50 PM, [EMAIL PROTECTED] <[EMAIL PROTECTED]
> >
> > >>> wrote:
> > >>>
> > >>>> Hi guys,
> > >>>>
> > >>>> I'm trying to maintain a bunch of simple kafka consumer to consume
> > >>> messages
> > >>>> from brokers. I know there is a way to send TopicMetadataRequest to
> > >>> broker
> > >>>> and get the response from the broker. But you have to specify the
> > >> broker
> > >>>> list to query the information. But broker might not be available
> > >> because