Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Is there a way to pull out kafka metadata from zookeeper?


Copy link to this message
-
Re: Is there a way to pull out kafka metadata from zookeeper?
What I suggested was *not* to get the metadata from the zookeeper, but
simply the broker list, to avoid having a list brokers on the configuration.

This seems reasonable for installations that don't have access to a VIP.
One option is to support broker list and zookeeper and get the broker list
from zookeeper if not directly configured. Can you file a JIRA to start
this discussion?

Thanks,
Neha
On Oct 12, 2013 5:18 AM, "Bruno D. Rodrigues" <[EMAIL PROTECTED]>
wrote:

> What I understood from Neha is that querying the metadata from one (any)
> kafka will return everything in one go. Querying the data indirectly via
> zookeeper would be more complicated and would involve more requests between
> the zookeeper and the brokers before being able to answer back.
>
> What I suggested was *not* to get the metadata from the zookeeper, but
> simply the broker list, to avoid having a list brokers on the
> configuration. Or more concretely, IMHO the producer and consumers should
> be consistent and allow either setting a list of zookeepers or a list of
> brokers.
>
> In both cases, independently of whatever action they need to do, they
> could ask a random broker for the information, or ask a random zookeeper
> for the list of brokers, and then a random broker.
>
> This for me would make it consistent and more professional. It would
> support using zookeeper or not. Would allow the developers to decide for
> themselves if they want to point to the brokers or to the zookeepers. But
> more importantly, if zookeeper is there to help the coordination of the
> brokers, the producers and consumers should rely on them to discover what
> they need to discover - the list of brokers.
>
> This way, one side pointing one way and the other side pointing a
> different way got me quite confused.
>
>
> A 12/10/2013, às 01:16, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> escreveu:
>
> > That's why I'm asking, I would like to see a kafka zookeeper client api
> to
> > get TopicMetadata instead of my own hacky way to query the zookeeper
> >
> > Thanks!
> > Best,
> > Siyuan
> >
> >
> > On Fri, Oct 11, 2013 at 4:00 PM, Bruno D. Rodrigues <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Why not ask zookeeper for the list of brokers and then ask a random
> >> broker for the metadata (and repeat if the broker is down), even if
> >> it's two calls.
> >>
> >> Heck it already does unnecessary connections. It connects to a broker,
> >> gets the metadata, disconnects, and then connects again for the data.
> >> If it's already assumed a producer or consumer will take some seconds
> >> until ready, what is another call gonna prejudice the flow.
> >>
> >> Then producers and consumers would then be consistently configured. Or
> >> allow the producers to also go to a broker instead of zookeeper.
> >>
> >> This way the consumer needs to know and hardcode at least one node.
> >> The node can fail. It can be changed.
> >>
> >> I thought zookeeper served to abstract this kind of complexity
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> --
> >> Bruno Rodrigues
> >> Sent from my iPhone
> >>
> >> No dia 11/10/2013, às 22:40, Neha Narkhede <[EMAIL PROTECTED]>
> >> escreveu:
> >>
> >>>>> For each consumer consumes different
> >>> topic/replica I have to specify those 20 brokers and go over all of
> them
> >> to
> >>> know which broker is alive. And even worse how about I dynamically add
> >> new
> >>> broker into the cluster and remove the old one
> >>>
> >>> TopicMetadataRequest is a batch API and you can get metadata
> information
> >>> for either a list of all topics or all topics in the cluster, if you
> >>> specify an empty list of topics. Adding a broker is not a problem since
> >> the
> >>> metadata request also returns the list of brokers in a cluster. The
> >> reason
> >>> this is better than reading from zookeeper is because the same
> operation
> >>> would require multiple zookeeper roundtrips, instead of a single
> >>> TopicMetadataRequest roundtrip to some kafka broker.