Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Suggestion on ZkClient usage in Kafka


+
Bae, Jae Hyeon 2012-11-19, 06:35
+
Sybrandy, Casey 2012-11-19, 13:31
+
Jun Rao 2012-11-19, 16:07
Copy link to this message
-
Re: Suggestion on ZkClient usage in Kafka
Are you saying that in 0.8, producers don't use zkclient?  Or don't need
it?  How can a producer dynamically respond to a change in the kafka
cluster without zk?

On Mon, Nov 19, 2012 at 8:07 AM, Jun Rao <[EMAIL PROTECTED]> wrote:

> Jae,
>
> In 0.8, producers don't need ZK client anymore. Instead, it uses a new
> getMetadata api to get topic/partition/leader information from the broker.
> Consumers still need ZK client. We plan to redesign the consumer post 0.8
> and can keep this in mind.
>
> Thanks,
>
> Jun
>
> On Sun, Nov 18, 2012 at 10:35 PM, Bae, Jae Hyeon <[EMAIL PROTECTED]>
> wrote:
>
> > I want to suggest kafka should create only one instance of ZkClient
> > globally because ZkClient is thread safe and it will make many users
> > easily customize kafka source code for Zookeeper.
> >
> > In our company's cloud environment, it is not recommended to create
> > ZkClient from zkConnect string directly because zookeeper cluster can
> > be dynamically changing. So, I have to create ZkClient using our
> > company's own platform library. Because of this requirement, I can't
> > use kafka jar file directly. I can modify and build kafka source code
> > but I have to repeat this work whenever I update kafka version, pretty
> > annoying.
> >
> > So, my suggestion is, let me pass ZkClient outs of Producer, Consumer,
> > and Broker, as the following example.
> >
> > Producer<String, String> producer > >
> >
> ProducerBuilder.withZkClient(zkClient).build<String,String>(producerConfig);
> >
> > ConsumerConnector connector > > Consumer.withZkClient(zkClient).createJavaConsumerConnector(new
> > ConsumerConfig(consumerProps));
> >
> > KafkaServer is a little more complicated but I believe without much
> > effort we can refactor KafkaServer to be customized with ZkClient.
> >
> > I really appreciate if this suggestion is accepted and merged to 0.8.
> > If you want me to contribute with this suggestion, please let me know
> > your opinion. If you are positive with this idea, I will contribute
> > very happily.
> >
> > Thank you
> > Best, Jae
> >
>
+
Neha Narkhede 2012-11-19, 20:20
+
Jason Rosenberg 2012-11-19, 21:29
+
Jun Rao 2012-11-20, 05:23
+
David Arthur 2012-11-20, 15:54
+
Jason Rosenberg 2012-11-20, 16:04
+
Jun Rao 2012-11-20, 16:15
+
Jun Rao 2012-11-20, 16:31
+
Bae, Jae Hyeon 2012-11-20, 17:58
+
Neha Narkhede 2012-11-20, 18:02
+
David Arthur 2012-11-20, 18:20
+
Jun Rao 2012-11-20, 18:42