Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Is there a way to pull out kafka metadata from zookeeper?


Copy link to this message
-
Re: Is there a way to pull out kafka metadata from zookeeper?
What I suggested was *not* to get the metadata from the zookeeper, but
simply the broker list, to avoid having a list brokers on the configuration.

This seems reasonable for installations that don't have access to a VIP.
One option is to support broker list and zookeeper and get the broker list
from zookeeper if not directly configured. Can you file a JIRA to start
this discussion?

Thanks,
Neha
On Oct 12, 2013 5:18 AM, "Bruno D. Rodrigues" <[EMAIL PROTECTED]>
wrote:

> What I understood from Neha is that querying the metadata from one (any)
> kafka will return everything in one go. Querying the data indirectly via
> zookeeper would be more complicated and would involve more requests between
> the zookeeper and the brokers before being able to answer back.
>
> What I suggested was *not* to get the metadata from the zookeeper, but
> simply the broker list, to avoid having a list brokers on the
> configuration. Or more concretely, IMHO the producer and consumers should
> be consistent and allow either setting a list of zookeepers or a list of
> brokers.
>
> In both cases, independently of whatever action they need to do, they
> could ask a random broker for the information, or ask a random zookeeper
> for the list of brokers, and then a random broker.
>
> This for me would make it consistent and more professional. It would
> support using zookeeper or not. Would allow the developers to decide for
> themselves if they want to point to the brokers or to the zookeepers. But
> more importantly, if zookeeper is there to help the coordination of the
> brokers, the producers and consumers should rely on them to discover what
> they need to discover - the list of brokers.
>
> This way, one side pointing one way and the other side pointing a
> different way got me quite confused.
>
>
> A 12/10/2013, às 01:16, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> escreveu:
>
> > That's why I'm asking, I would like to see a kafka zookeeper client api
> to
> > get TopicMetadata instead of my own hacky way to query the zookeeper
> >
> > Thanks!
> > Best,
> > Siyuan
> >
> >
> > On Fri, Oct 11, 2013 at 4:00 PM, Bruno D. Rodrigues <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Why not ask zookeeper for the list of brokers and then ask a random
> >> broker for the metadata (and repeat if the broker is down), even if
> >> it's two calls.
> >>
> >> Heck it already does unnecessary connections. It connects to a broker,
> >> gets the metadata, disconnects, and then connects again for the data.
> >> If it's already assumed a producer or consumer will take some seconds
> >> until ready, what is another call gonna prejudice the flow.
> >>
> >> Then producers and consumers would then be consistently configured. Or
> >> allow the producers to also go to a broker instead of zookeeper.
> >>
> >> This way the consumer needs to know and hardcode at least one node.
> >> The node can fail. It can be changed.
> >>
> >> I thought zookeeper served to abstract this kind of complexity
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> --
> >> Bruno Rodrigues
> >> Sent from my iPhone
> >>
> >> No dia 11/10/2013, às 22:40, Neha Narkhede <[EMAIL PROTECTED]>
> >> escreveu:
> >>
> >>>>> For each consumer consumes different
> >>> topic/replica I have to specify those 20 brokers and go over all of
> them
> >> to
> >>> know which broker is alive. And even worse how about I dynamically add
> >> new
> >>> broker into the cluster and remove the old one
> >>>
> >>> TopicMetadataRequest is a batch API and you can get metadata
> information
> >>> for either a list of all topics or all topics in the cluster, if you
> >>> specify an empty list of topics. Adding a broker is not a problem since
> >> the
> >>> metadata request also returns the list of brokers in a cluster. The
> >> reason
> >>> this is better than reading from zookeeper is because the same
> operation
> >>> would require multiple zookeeper roundtrips, instead of a single
> >>> TopicMetadataRequest roundtrip to some kafka broker.

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB