Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # dev >> Re: Get all the partion details from zookeeper for a topic


+
sourabh chaki 2013-02-27, 12:33
+
Neha Narkhede 2013-02-27, 17:52
+
sourabh chaki 2013-03-05, 14:48
+
Neha Narkhede 2013-03-05, 16:31
+
sourabh chaki 2013-03-06, 07:05
+
Neha Narkhede 2013-03-06, 16:51
+
sourabh chaki 2013-03-06, 19:43
Copy link to this message
-
Re: Get all the partion details from zookeeper for a topic
You can instantiate ZkClient with the ZkStringSerializer in Kafka -
      zkClient = new ZkClient(params.zkConnect, 30000, 30000,
ZKStringSerializer)

For code examples on how to read the various paths, you can take a look at
kafka.utils.ZkUtils.scala

Thanks,
Neha
On Wed, Mar 6, 2013 at 11:43 AM, sourabh chaki <[EMAIL PROTECTED]>wrote:

> Thanks Neha for your response.
> Now I am calling it like:
> List<String> partitions =
> zkclient.readData("/brokers/topics/test-topic/1");
> //topic = test topic and broker id is 1
>
> *1.  *     Now _connection.readData(path, stat, watch); is returning a
> byte[] of [49]. But derializable(data) is  throwing ZkMarshallingError.
>
> org.I0Itec.zkclient.exception.ZkMarshallingError: java.io.EOFException
> at
>
> org.I0Itec.zkclient.serialize.SerializableSerializer.deserialize(SerializableSerializer.java:37)
>
> *2.*       Apart from that byte 49 is number 1. But here my partition id is
> 0. Is derializable(data)  suppose to convert [49] to [0] ?
>
>
> Code snippet where it is failing.
> Class ZkClient.java
>
>  protected <T extends Object> T readData(final String path, final Stat
> stat, final boolean watch) {
>         byte[] data = retryUntilConnected(new Callable<byte[]>() {
>
>             @Override
>             public byte[] call() throws Exception {
>                 return _connection.readData(path, stat, watch);
>             }
>         });
>         return (T) derializable(data);   //data here [49]. But derializable
> throwing ZkMarshallingError
>     }
>
> Please help.
>
> Thanks
> Sourabh
>
>
> On Wed, Mar 6, 2013 at 10:20 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > I didn't check before I typed. I was referring to readData().
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, Mar 5, 2013 at 11:04 PM, sourabh chaki <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi Neha,
> > > I am using zkclient version 0.1. I dont see any method "getData" in
> > > zkclient. Please let me know which version should I use.
> > >
> > > Thanks
> > > Sourabh
> > >
> > >
> > > On Tue, Mar 5, 2013 at 10:01 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > The problem is that you need to invoke getData, not getChildren on
> this
> > > > path - /brokers/topics/test-topic/1. That will return the # of
> > partitions
> > > > on this broker, which is 1.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Hi Neha,
> > > > >
> > > > > Thanks for your reply.
> > > > >
> > > > > ZkClient zk = new ZkClient(serverstring );
> > > > > List<String> partitions =
> > > zk.getChildren("/brokers/topics/test-topic/1");
> > > > > // my topic = test-topic, broker id = 1
> > > > >
> > > > > This broker is associated with partition 0. So I am expecting
> output
> > > > should
> > > > > be [0] but I am getting empty response [].
> > > > >
> > > > > Is there anything wrong in the way I am calling it?
> > > > > Please help.
> > > > >
> > > > > Thanks
> > > > > Sourabh
> > > > >
> > > > >
> > > > > On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <
> > > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > Sourabh,
> > > > > >
> > > > > > In 0.7, you can use zkclient or the zookeeper client library to
> > query
> > > > the
> > > > > > following zookeeper path to get a number of partitions for a
> topic
> > -
> > > > > >
> > > > > > get /brokers/topics/[topic]/[broker-id]
> > > > > >
> > > > > > This will give you # of partitions/broker, you can then add up
> this
> > > > value
> > > > > > across all brokers.
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <
> > > > [EMAIL PROTECTED]
> > > > > > >wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I am using zookeeper based connector to get data from Kafka
> > Server.
> > > > Is
> > > > > > > there any way to get all the partitions from zookeeper for a