Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # dev >> Re: Get all the partion details from zookeeper for a topic


Copy link to this message
-
Re: Get all the partion details from zookeeper for a topic
You can instantiate ZkClient with the ZkStringSerializer in Kafka -
      zkClient = new ZkClient(params.zkConnect, 30000, 30000,
ZKStringSerializer)

For code examples on how to read the various paths, you can take a look at
kafka.utils.ZkUtils.scala

Thanks,
Neha
On Wed, Mar 6, 2013 at 11:43 AM, sourabh chaki <[EMAIL PROTECTED]>wrote:

> Thanks Neha for your response.
> Now I am calling it like:
> List<String> partitions =
> zkclient.readData("/brokers/topics/test-topic/1");
> //topic = test topic and broker id is 1
>
> *1.  *     Now _connection.readData(path, stat, watch); is returning a
> byte[] of [49]. But derializable(data) is  throwing ZkMarshallingError.
>
> org.I0Itec.zkclient.exception.ZkMarshallingError: java.io.EOFException
> at
>
> org.I0Itec.zkclient.serialize.SerializableSerializer.deserialize(SerializableSerializer.java:37)
>
> *2.*       Apart from that byte 49 is number 1. But here my partition id is
> 0. Is derializable(data)  suppose to convert [49] to [0] ?
>
>
> Code snippet where it is failing.
> Class ZkClient.java
>
>  protected <T extends Object> T readData(final String path, final Stat
> stat, final boolean watch) {
>         byte[] data = retryUntilConnected(new Callable<byte[]>() {
>
>             @Override
>             public byte[] call() throws Exception {
>                 return _connection.readData(path, stat, watch);
>             }
>         });
>         return (T) derializable(data);   //data here [49]. But derializable
> throwing ZkMarshallingError
>     }
>
> Please help.
>
> Thanks
> Sourabh
>
>
> On Wed, Mar 6, 2013 at 10:20 PM, Neha Narkhede <[EMAIL PROTECTED]
> >wrote:
>
> > I didn't check before I typed. I was referring to readData().
> >
> > Thanks,
> > Neha
> >
> >
> > On Tue, Mar 5, 2013 at 11:04 PM, sourabh chaki <[EMAIL PROTECTED]
> > >wrote:
> >
> > > Hi Neha,
> > > I am using zkclient version 0.1. I dont see any method "getData" in
> > > zkclient. Please let me know which version should I use.
> > >
> > > Thanks
> > > Sourabh
> > >
> > >
> > > On Tue, Mar 5, 2013 at 10:01 PM, Neha Narkhede <
> [EMAIL PROTECTED]
> > > >wrote:
> > >
> > > > The problem is that you need to invoke getData, not getChildren on
> this
> > > > path - /brokers/topics/test-topic/1. That will return the # of
> > partitions
> > > > on this broker, which is 1.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > >
> > > > On Tue, Mar 5, 2013 at 6:48 AM, sourabh chaki <
> [EMAIL PROTECTED]
> > > > >wrote:
> > > >
> > > > > Hi Neha,
> > > > >
> > > > > Thanks for your reply.
> > > > >
> > > > > ZkClient zk = new ZkClient(serverstring );
> > > > > List<String> partitions =
> > > zk.getChildren("/brokers/topics/test-topic/1");
> > > > > // my topic = test-topic, broker id = 1
> > > > >
> > > > > This broker is associated with partition 0. So I am expecting
> output
> > > > should
> > > > > be [0] but I am getting empty response [].
> > > > >
> > > > > Is there anything wrong in the way I am calling it?
> > > > > Please help.
> > > > >
> > > > > Thanks
> > > > > Sourabh
> > > > >
> > > > >
> > > > > On Wed, Feb 27, 2013 at 11:22 PM, Neha Narkhede <
> > > [EMAIL PROTECTED]
> > > > > >wrote:
> > > > >
> > > > > > Sourabh,
> > > > > >
> > > > > > In 0.7, you can use zkclient or the zookeeper client library to
> > query
> > > > the
> > > > > > following zookeeper path to get a number of partitions for a
> topic
> > -
> > > > > >
> > > > > > get /brokers/topics/[topic]/[broker-id]
> > > > > >
> > > > > > This will give you # of partitions/broker, you can then add up
> this
> > > > value
> > > > > > across all brokers.
> > > > > >
> > > > > > Thanks,
> > > > > > Neha
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 27, 2013 at 4:32 AM, sourabh chaki <
> > > > [EMAIL PROTECTED]
> > > > > > >wrote:
> > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > I am using zookeeper based connector to get data from Kafka
> > Server.
> > > > Is
> > > > > > > there any way to get all the partitions from zookeeper for a

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB