-Re: Is there a way to pull out kafka metadata from zookeeper?
Neha Narkhede 2013-10-12, 18:02
What I suggested was *not* to get the metadata from the zookeeper, but
simply the broker list, to avoid having a list brokers on the configuration.
This seems reasonable for installations that don't have access to a VIP.
One option is to support broker list and zookeeper and get the broker list
from zookeeper if not directly configured. Can you file a JIRA to start
On Oct 12, 2013 5:18 AM, "Bruno D. Rodrigues" <[EMAIL PROTECTED]>
> What I understood from Neha is that querying the metadata from one (any)
> kafka will return everything in one go. Querying the data indirectly via
> zookeeper would be more complicated and would involve more requests between
> the zookeeper and the brokers before being able to answer back.
> What I suggested was *not* to get the metadata from the zookeeper, but
> simply the broker list, to avoid having a list brokers on the
> configuration. Or more concretely, IMHO the producer and consumers should
> be consistent and allow either setting a list of zookeepers or a list of
> In both cases, independently of whatever action they need to do, they
> could ask a random broker for the information, or ask a random zookeeper
> for the list of brokers, and then a random broker.
> This for me would make it consistent and more professional. It would
> support using zookeeper or not. Would allow the developers to decide for
> themselves if they want to point to the brokers or to the zookeepers. But
> more importantly, if zookeeper is there to help the coordination of the
> brokers, the producers and consumers should rely on them to discover what
> they need to discover - the list of brokers.
> This way, one side pointing one way and the other side pointing a
> different way got me quite confused.
> A 12/10/2013, às 01:16, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> escreveu:
> > That's why I'm asking, I would like to see a kafka zookeeper client api
> > get TopicMetadata instead of my own hacky way to query the zookeeper
> > Thanks!
> > Best,
> > Siyuan
> > On Fri, Oct 11, 2013 at 4:00 PM, Bruno D. Rodrigues <
> > [EMAIL PROTECTED]> wrote:
> >> Why not ask zookeeper for the list of brokers and then ask a random
> >> broker for the metadata (and repeat if the broker is down), even if
> >> it's two calls.
> >> Heck it already does unnecessary connections. It connects to a broker,
> >> gets the metadata, disconnects, and then connects again for the data.
> >> If it's already assumed a producer or consumer will take some seconds
> >> until ready, what is another call gonna prejudice the flow.
> >> Then producers and consumers would then be consistently configured. Or
> >> allow the producers to also go to a broker instead of zookeeper.
> >> This way the consumer needs to know and hardcode at least one node.
> >> The node can fail. It can be changed.
> >> I thought zookeeper served to abstract this kind of complexity
> >> --
> >> Bruno Rodrigues
> >> Sent from my iPhone
> >> No dia 11/10/2013, às 22:40, Neha Narkhede <[EMAIL PROTECTED]>
> >> escreveu:
> >>>>> For each consumer consumes different
> >>> topic/replica I have to specify those 20 brokers and go over all of
> >> to
> >>> know which broker is alive. And even worse how about I dynamically add
> >> new
> >>> broker into the cluster and remove the old one
> >>> TopicMetadataRequest is a batch API and you can get metadata
> >>> for either a list of all topics or all topics in the cluster, if you
> >>> specify an empty list of topics. Adding a broker is not a problem since
> >> the
> >>> metadata request also returns the list of brokers in a cluster. The
> >> reason
> >>> this is better than reading from zookeeper is because the same
> >>> would require multiple zookeeper roundtrips, instead of a single
> >>> TopicMetadataRequest roundtrip to some kafka broker.