Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Consumer group and Partitions


Copy link to this message
-
Re: Consumer group and Partitions
That should be all you need - although you only need to bounce the kafka
broker. Are you sending enough messages? If you send only a few test
messages, it is possible for the messages to get sent to only one
partition. E.g., I tried a quick test with the ConsoleProducer and it took
three messages before the partition was created.

Joel

On Mon, Jun 25, 2012 at 8:43 AM, Narala, Ravi <[EMAIL PROTECTED]>wrote:

> Jun ,
> Thanks for the quick repsonse. I am running into a partition issue. I am
> not sure if am doing the right way. Here are the steps I did to create the
> partition. But I still do not see them partitioned.
>
> Stopped the kafka and zookeeper server
> Modified the num.partitions entry in server.properties from 1 to 2.
> Restarted the zookeeper and kafka server
> Used a Producer client to send messages to a topic named
> 'poc_test_topic1'.(I did not specify any partition key)
>
> The logs dir was configured to /tmp/kafka-logs in server.properties.
>
> I see only a 'poc_test_topic1-0' topic under logs dir. I was expecting to
> see 'poc_test_topic1-0' and 'poc_test_topic1-1' partitions under it.
>
> Am I missing something or is this all I need to create a partition ?
>
> Ravi Narala
> Web Services & Integration
> Level 3 Communications
> 200 Office Galleria Center
> Southfield, MI 48035 US
> p: 248-436-9324
> e: [EMAIL PROTECTED]
>
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Monday, June 25, 2012 11:05 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Consumer group and Partitions
>
> Ravi,
>
> For 1, consumer group is specified in consumer config.
>
> For 2, partitions are divided evenly among consumers in the same group for
> consumption. For details, please see the design doc on our website.
>
> For 3, a producer never needs to know consumer groups. A producer always
> produces messages to a topic/partition. There are examples in the design
> doc too.
>
> Thanks,
>
> Jun
>
> On Mon, Jun 25, 2012 at 7:57 AM, Narala, Ravi <[EMAIL PROTECTED]
> >wrote:
>
> > Hi,
> > I have couple of questions regarding the concepts of Consumer group
> > and partitions. Here are those ?
> >
> >
> > 1.       When consuming a message how Do I know which consumer group I
> > belong to?. Is there a configuration available or do we have to
> > specify it at the application level ?
> >
> > 2.       How do consumer groups work in conjunction with partitions ?
> >
> > 3.       How do I produce a message to specific group and a partition ?.
> A
> > coding example is greatly appreciated.
> >
> >
> >
> > Thanks,
> >
> > Ravi Narala
> > Web Services & Integration
> > Level 3 Communications
> > 200 Office Galleria Center
> > Southfield, MI 48035 US
> > p: 248-436-9324
> > e: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
> >
> >
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB