Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Consumer group and Partitions


Copy link to this message
-
Re: Consumer group and Partitions
Joel Koshy 2012-06-25, 17:47
That should be all you need - although you only need to bounce the kafka
broker. Are you sending enough messages? If you send only a few test
messages, it is possible for the messages to get sent to only one
partition. E.g., I tried a quick test with the ConsoleProducer and it took
three messages before the partition was created.

Joel

On Mon, Jun 25, 2012 at 8:43 AM, Narala, Ravi <[EMAIL PROTECTED]>wrote:

> Jun ,
> Thanks for the quick repsonse. I am running into a partition issue. I am
> not sure if am doing the right way. Here are the steps I did to create the
> partition. But I still do not see them partitioned.
>
> Stopped the kafka and zookeeper server
> Modified the num.partitions entry in server.properties from 1 to 2.
> Restarted the zookeeper and kafka server
> Used a Producer client to send messages to a topic named
> 'poc_test_topic1'.(I did not specify any partition key)
>
> The logs dir was configured to /tmp/kafka-logs in server.properties.
>
> I see only a 'poc_test_topic1-0' topic under logs dir. I was expecting to
> see 'poc_test_topic1-0' and 'poc_test_topic1-1' partitions under it.
>
> Am I missing something or is this all I need to create a partition ?
>
> Ravi Narala
> Web Services & Integration
> Level 3 Communications
> 200 Office Galleria Center
> Southfield, MI 48035 US
> p: 248-436-9324
> e: [EMAIL PROTECTED]
>
> -----Original Message-----
> From: Jun Rao [mailto:[EMAIL PROTECTED]]
> Sent: Monday, June 25, 2012 11:05 AM
> To: [EMAIL PROTECTED]
> Subject: Re: Consumer group and Partitions
>
> Ravi,
>
> For 1, consumer group is specified in consumer config.
>
> For 2, partitions are divided evenly among consumers in the same group for
> consumption. For details, please see the design doc on our website.
>
> For 3, a producer never needs to know consumer groups. A producer always
> produces messages to a topic/partition. There are examples in the design
> doc too.
>
> Thanks,
>
> Jun
>
> On Mon, Jun 25, 2012 at 7:57 AM, Narala, Ravi <[EMAIL PROTECTED]
> >wrote:
>
> > Hi,
> > I have couple of questions regarding the concepts of Consumer group
> > and partitions. Here are those ?
> >
> >
> > 1.       When consuming a message how Do I know which consumer group I
> > belong to?. Is there a configuration available or do we have to
> > specify it at the application level ?
> >
> > 2.       How do consumer groups work in conjunction with partitions ?
> >
> > 3.       How do I produce a message to specific group and a partition ?.
> A
> > coding example is greatly appreciated.
> >
> >
> >
> > Thanks,
> >
> > Ravi Narala
> > Web Services & Integration
> > Level 3 Communications
> > 200 Office Galleria Center
> > Southfield, MI 48035 US
> > p: 248-436-9324
> > e: [EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>
> >
> >
>