Hi All,

I am using zookeeper based connector to get data from Kafka Server. Is
there any way to get all the partitions from zookeeper for a given topic?

I need do do that through java code.

Please help!!!

Thanks.

 

Sourabh,

In 0.7, you can use zkclient or the zookeeper client library to query the
following zookeeper path to get a number of partitions for a topic -

get /brokers/topics/[topic]/[broker-id]

This will give you # of partitions/broker, you can then add up this value
across all brokers.

Thanks,
Neha
On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <[EMAIL PROTECTED]>wrote:

> Hi All,
>
> I am using zookeeper based connector to get data from Kafka Server. Is
> there any way to get all the partitions from zookeeper for a given topic?
>
> I need do do that through java code.
>
> Please help!!!
>
> Thanks.
>

 

Hi Neha,

Thanks for your reply.

ZkClient zk = new ZkClient(serverstring );
List<String> partitions = zk.getChildren("/brokers/topics/test-topic/1");
// my topic = test-topic, broker id = 1

This broker is associated with partition 0. So I am expecting output should
be [0] but I am getting empty response [].

Is there anything wrong in the way I am calling it?
Please help.

Thanks
Sourabh
On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> Sourabh,
>
> In 0.7, you can use zkclient or the zookeeper client library to query the
> following zookeeper path to get a number of partitions for a topic -
>
> get /brokers/topics/[topic]/[broker-id]
>
> This will give you # of partitions/broker, you can then add up this value
> across all brokers.
>
> Thanks,
> Neha
>
>
> On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <[EMAIL PROTECTED]
> >wrote:
>
> > Hi All,
> >
> > I am using zookeeper based connector to get data from Kafka Server. Is
> > there any way to get all the partitions from zookeeper for a given topic?
> >
> > I need do do that through java code.
> >
> > Please help!!!
> >
> > Thanks.
> >
>

 

The problem is that you need to invoke getData, not getChildren on this
path - /brokers/topics/test-topic/1. That will return the # of partitions
on this broker, which is 1.

Thanks,
Neha
On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki <[EMAIL PROTECTED]>wrote:

> Hi Neha,
>
> Thanks for your reply.
>
> ZkClient zk = new ZkClient(serverstring );
> List<String> partitions = zk.getChildren("/brokers/topics/test-topic/1");
> // my topic = test-topic, broker id = 1
>
> This broker is associated with partition 0. So I am expecting output should
> be [0] but I am getting empty response [].
>
> Is there anything wrong in the way I am calling it?
> Please help.
>
> Thanks
> Sourabh
>
>
> On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > Sourabh,
> >
> > In 0.7, you can use zkclient or the zookeeper client library to query the
> > following zookeeper path to get a number of partitions for a topic -
> >
> > get /brokers/topics/[topic]/[broker-id]
> >
> > This will give you # of partitions/broker, you can then add up this value
> > across all brokers.
> >
> > Thanks,
> > Neha
> >
> >
> > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi All,
> > >
> > > I am using zookeeper based connector to get data from Kafka Server. Is
> > > there any way to get all the partitions from zookeeper for a given
> topic?
> > >
> > > I need do do that through java code.
> > >
> > > Please help!!!
> > >
> > > Thanks.
> > >
> >
>

 

Hi Neha,
I am using zkclient version 0.1. I dont see any method "getData" in
zkclient. Please let me know which version should I use.

Thanks
Sourabh
On Tue, Mar 5, 2013 at 10:01 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> The problem is that you need to invoke getData, not getChildren on this
> path - /brokers/topics/test-topic/1. That will return the # of partitions
> on this broker, which is 1.
>
> Thanks,
> Neha
>
>
> On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Neha,
> >
> > Thanks for your reply.
> >
> > ZkClient zk = new ZkClient(serverstring );
> > List<String> partitions = zk.getChildren("/brokers/topics/test-topic/1");
> > // my topic = test-topic, broker id = 1
> >
> > This broker is associated with partition 0. So I am expecting output
> should
> > be [0] but I am getting empty response [].
> >
> > Is there anything wrong in the way I am calling it?
> > Please help.
> >
> > Thanks
> > Sourabh
> >
> >
> > On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Sourabh,
> > >
> > > In 0.7, you can use zkclient or the zookeeper client library to query
> the
> > > following zookeeper path to get a number of partitions for a topic -
> > >
> > > get /brokers/topics/[topic]/[broker-id]
> > >
> > > This will give you # of partitions/broker, you can then add up this
> value
> > > across all brokers.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Hi All,
> > > >
> > > > I am using zookeeper based connector to get data from Kafka Server.
> Is
> > > > there any way to get all the partitions from zookeeper for a given
> > topic?
> > > >
> > > > I need do do that through java code.
> > > >
> > > > Please help!!!
> > > >
> > > > Thanks.
> > > >
> > >
> >
>

 

I didn't check before I typed. I was referring to readData().

Thanks,
Neha
On Tue, Mar 5, 2013 at 11:04 PM, sourabh chaki <[EMAIL PROTECTED]>wrote:

> Hi Neha,
> I am using zkclient version 0.1. I dont see any method "getData" in
> zkclient. Please let me know which version should I use.
>
> Thanks
> Sourabh
>
>
> On Tue, Mar 5, 2013 at 10:01 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > The problem is that you need to invoke getData, not getChildren on this
> > path - /brokers/topics/test-topic/1. That will return the # of partitions
> > on this broker, which is 1.
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi Neha,
> > >
> > > Thanks for your reply.
> > >
> > > ZkClient zk = new ZkClient(serverstring );
> > > List<String> partitions =
> zk.getChildren("/brokers/topics/test-topic/1");
> > > // my topic = test-topic, broker id = 1
> > >
> > > This broker is associated with partition 0. So I am expecting output
> > should
> > > be [0] but I am getting empty response [].
> > >
> > > Is there anything wrong in the way I am calling it?
> > > Please help.
> > >
> > > Thanks
> > > Sourabh
> > >
> > >
> > > On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Sourabh,
> > > >
> > > > In 0.7, you can use zkclient or the zookeeper client library to query
> > the
> > > > following zookeeper path to get a number of partitions for a topic -
> > > >
> > > > get /brokers/topics/[topic]/[broker-id]
> > > >
> > > > This will give you # of partitions/broker, you can then add up this
> > value
> > > > across all brokers.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I am using zookeeper based connector to get data from Kafka Server.
> > Is
> > > > > there any way to get all the partitions from zookeeper for a given
> > > topic?
> > > > >
> > > > > I need do do that through java code.
> > > > >
> > > > > Please help!!!
> > > > >
> > > > > Thanks.
> > > > >
> > > >
> > >
> >
>

 

Thanks Neha for your response.
Now I am calling it like:
List<String> partitions = zkclient.readData("/brokers/topics/test-topic/1");
//topic = test topic and broker id is 1

*1.  *     Now _connection.readData(path, stat, watch); is returning a
byte[] of [49]. But derializable(data) is  throwing ZkMarshallingError.

org.I0Itec.zkclient.exception.ZkMarshallingError: java.io.EOFException
at
org.I0Itec.zkclient.serialize.SerializableSerializer.deserialize(SerializableSerializer.java:37)

*2.*       Apart from that byte 49 is number 1. But here my partition id is
0. Is derializable(data)  suppose to convert [49] to [0] ?
Code snippet where it is failing.
Class ZkClient.java

 protected <T extends Object> T readData(final String path, final Stat
stat, final boolean watch) {
        byte[] data = retryUntilConnected(new Callable<byte[]>() {

            @Override
            public byte[] call() throws Exception {
                return _connection.readData(path, stat, watch);
            }
        });
        return (T) derializable(data);   //data here [49]. But derializable
throwing ZkMarshallingError
    }

Please help.

Thanks
Sourabh
On Wed, Mar 6, 2013 at 10:20 PM, Neha Narkhede <[EMAIL PROTECTED]>wrote:

> I didn't check before I typed. I was referring to readData().
>
> Thanks,
> Neha
>
>
> On Tue, Mar 5, 2013 at 11:04 PM, sourabh chaki <[EMAIL PROTECTED]
> >wrote:
>
> > Hi Neha,
> > I am using zkclient version 0.1. I dont see any method "getData" in
> > zkclient. Please let me know which version should I use.
> >
> > Thanks
> > Sourabh
> >
> >
> > On Tue, Mar 5, 2013 at 10:01 PM, Neha Narkhede <[EMAIL PROTECTED]
> > >wrote:
> >
> > > The problem is that you need to invoke getData, not getChildren on this
> > > path - /brokers/topics/test-topic/1. That will return the # of
> partitions
> > > on this broker, which is 1.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki <[EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > Hi Neha,
> > > >
> > > > Thanks for your reply.
> > > >
> > > > ZkClient zk = new ZkClient(serverstring );
> > > > List<String> partitions =
> > zk.getChildren("/brokers/topics/test-topic/1");
> > > > // my topic = test-topic, broker id = 1
> > > >
> > > > This broker is associated with partition 0. So I am expecting output
> > > should
> > > > be [0] but I am getting empty response [].
> > > >
> > > > Is there anything wrong in the way I am calling it?
> > > > Please help.
> > > >
> > > > Thanks
> > > > Sourabh
> > > >
> > > >
> > > > On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Sourabh,
> > > > >
> > > > > In 0.7, you can use zkclient or the zookeeper client library to
> query
> > > the
> > > > > following zookeeper path to get a number of partitions for a topic
> -
> > > > >
> > > > > get /brokers/topics/[topic]/[broker-id]
> > > > >
> > > > > This will give you # of partitions/broker, you can then add up this
> > > value
> > > > > across all brokers.
> > > > >
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > > >
> > > > > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <
> > > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I am using zookeeper based connector to get data from Kafka
> Server.
> > > Is
> > > > > > there any way to get all the partitions from zookeeper for a
> given
> > > > topic?
> > > > > >
> > > > > > I need do do that through java code.
> > > > > >
> > > > > > Please help!!!
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > >
> > > >
> > >
> >
>

 

You can instantiate ZkClient with the ZkStringSerializer in Kafka -
      zkClient = new ZkClient(params.zkConnect, 30000, 30000,
ZKStringSerializer)

For code examples on how to read the various paths, you can take a look at
kafka.utils.ZkUtils.scala

Thanks,
Neha
On Wed, Mar 6, 2013 at 11:43 AM, sourabh chaki <[EMAIL PROTECTED]>wrote:

> Thanks Neha for your response.
> Now I am calling it like:
> List<String> partitions =
> zkclient.readData("/brokers/topics/test-topic/1");
> //topic = test topic and broker id is 1
>
> *1.  *     Now _connection.readData(path, stat, watch); is returning a
> byte[] of [49]. But derializable(data) is  throwing ZkMarshallingError.
>
> org.I0Itec.zkclient.exception.ZkMarshallingError: java.io.EOFException
> at
>
> org.I0Itec.zkclient.serialize.SerializableSerializer.deserialize(SerializableSerializer.java:37)
>
> *2.*       Apart from that byte 49 is number 1. But here my partition id is
> 0. Is derializable(data)  suppose to convert [49] to [0] ?
>
>
> Code snippet where it is failing.
> Class ZkClient.java
>
>  protected <T extends Object> T readData(final String path, final Stat
> stat, final boolean watch) {
>         byte[] data = retryUntilConnected(new Callable<byte[]>() {
>
>             @Override
>             public byte[] call() throws Exception {
>                 return _connection.readData(path, stat, watch);
>             }
>         });
>         return (T) derializable(data);   //data here [49]. But derializable
> throwing ZkMarshallingError
>     }
>
> Please help.
>
> Thanks
> Sourabh
>
>
> On Wed, Mar 6, 2013 at 10:20 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > I didn't check before I typed. I was referring to readData().
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, Mar 5, 2013 at 11:04 PM, sourabh chaki <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi Neha,
> > > I am using zkclient version 0.1. I dont see any method "getData" in
> > > zkclient. Please let me know which version should I use.
> > >
> > > Thanks
> > > Sourabh
> > >
> > >
> > > On Tue, Mar 5, 2013 at 10:01 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > The problem is that you need to invoke getData, not getChildren on
> this
> > > > path - /brokers/topics/test-topic/1. That will return the # of
> > partitions
> > > > on this broker, which is 1.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Hi Neha,
> > > > >
> > > > > Thanks for your reply.
> > > > >
> > > > > ZkClient zk = new ZkClient(serverstring );
> > > > > List<String> partitions =
> > > zk.getChildren("/brokers/topics/test-topic/1");
> > > > > // my topic = test-topic, broker id = 1
> > > > >
> > > > > This broker is associated with partition 0. So I am expecting
> output
> > > > should
> > > > > be [0] but I am getting empty response [].
> > > > >
> > > > > Is there anything wrong in the way I am calling it?
> > > > > Please help.
> > > > >
> > > > > Thanks
> > > > > Sourabh
> > > > >
> > > > >
> > > > > On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <
> > > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > Sourabh,
> > > > > >
> > > > > > In 0.7, you can use zkclient or the zookeeper client library to
> > query
> > > > the
> > > > > > following zookeeper path to get a number of partitions for a
> topic
> > -
> > > > > >
> > > > > > get /brokers/topics/[topic]/[broker-id]
> > > > > >
> > > > > > This will give you # of partitions/broker, you can then add up
> this
> > > > value
> > > > > > across all brokers.
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <
> > > > [EMAIL PROTECTED]
> > > > > > >wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I am using zookeeper based connector to get data from Kafka
> > Server.
> > > > Is
> > > > > > > there any way to get all the partitions from zookeeper for a