Did you get my previous answer? I am including it below.

Currently, partition is the smallest unit that we distribute data among
consumers (in the same consumer group). So, if the # of consumers is larger
than the total number of partitions in a Kafka cluster (across all
brokers), some consumers will never get any data.

Thanks,

Jun

On Wed, Dec 19, 2012 at 3:07 AM, 永辉 赵 <[EMAIL PROTECTED]> wrote:

>
> Hi all,
>
> In consumer we use a topic count map to set how many stream we want to
> split
> the topic.
> What's the relation between kafka partition number and consumer stream
> number?
> Seems  stream number should be less or equal to partition number?
> Because if we set stream number bigger than partition number then some
> stream is empty or some partition is split.
>
> Thanks,
> Yonghui
>
>
>
>

 

Thanks Jun, I didn't get you previous answer.
I think I didn't state my question clearly.

I know the number of the consumer in one consumer group should be less or equal than partition number.
But my question is about the relation between the stream number in one consumer and partition number subscribed by this consumer.  

For example this piece of code create only one stream in the consumer, so all partitions data will go into one stream?

    Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
    topicCountMap.put(topic, new Integer(1));
    Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
    KafkaStream<byte[], byte[]> stream = consumerMap.get(topic).get(0);
    ConsumerIterator<byte[], byte[]> it = stream.iterator();
  
Sent from my iPad

ÔÚ 2012-12-19£¬23:49£¬Jun Rao <[EMAIL PROTECTED]> дµÀ£º

> Did you get my previous answer? I am including it below.
>
> Currently, partition is the smallest unit that we distribute data among consumers (in the same consumer group). So, if the # of consumers is larger than the total number of partitions in a Kafka cluster (across all brokers), some consumers will never get any data.
>
> Thanks,
>
> Jun
>
> On Wed, Dec 19, 2012 at 3:07 AM, ÓÀ»Ô ÕÔ <[EMAIL PROTECTED]> wrote:
>
> Hi all,
>
> In consumer we use a topic count map to set how many stream we want to split
> the topic.
> What's the relation between kafka partition number and consumer stream
> number?
> Seems  stream number should be less or equal to partition number?
> Because if we set stream number bigger than partition number then some
> stream is empty or some partition is split.
>
> Thanks,
> Yonghui
>
>
>
>
 

In your example, yes, all partitions go to one stream. In general, each
stream maps 1 or more partitions.

Thanks,

Jun

On Wed, Dec 19, 2012 at 8:32 AM, Yonghui Zhao <[EMAIL PROTECTED]> wrote:

> Thanks Jun, I didn't get you previous answer.
> I think I didn't state my question clearly.
>
> I know the number of the consumer in one consumer group should be less or
> equal than partition number.
> But my question is about the relation between the stream number in one
> consumer and partition number subscribed by this consumer.
>
> For example this piece of code create only one stream in the consumer, so
> all partitions data will go into one stream?
>
>     Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
>     topicCountMap.put(topic, new Integer(1));
>     Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
>     KafkaStream<byte[], byte[]> stream =  consumerMap.get(topic).get(0);
>     ConsumerIterator<byte[], byte[]> it = stream.iterator();
>
>
>
> Sent from my iPad
>
> 在 2012-12-19，23:49，Jun Rao <[EMAIL PROTECTED]> 写道：
>
> Did you get my previous answer? I am including it below.
>
> Currently, partition is the smallest unit that we distribute data among
> consumers (in the same consumer group). So, if the # of consumers is larger
> than the total number of partitions in a Kafka cluster (across all
> brokers), some consumers will never get any data.
>
> Thanks,
>
> Jun
>
> On Wed, Dec 19, 2012 at 3:07 AM, 永辉 赵 <[EMAIL PROTECTED]> wrote:
>
>>
>> Hi all,
>>
>> In consumer we use a topic count map to set how many stream we want to
>> split
>> the topic.
>> What's the relation between kafka partition number and consumer stream
>> number?
>> Seems  stream number should be less or equal to partition number?
>> Because if we set stream number bigger than partition number then some
>> stream is empty or some partition is split.
>>
>> Thanks,
>> Yonghui
>>
>>
>>
>>
>

 

If there is no partitions (only one partition), then two consumers exist.
Is it possible only one consumer  to get data from kafka(two broker)?
I can`t understand ...

-----Original Message-----
From: Jun Rao [mailto:[EMAIL PROTECTED]]
Sent: Tuesday, December 18, 2012 1:39 AM
To: [EMAIL PROTECTED]
Subject: Re: partition and consumer stream number

Currently, partition is the smallest unit that we distribute data among consumers (in the same consumer group). So, if the # of consumers is larger than the total number of partitions in a Kafka cluster (across all brokers), some consumers will never get any data.

Thanks,

Jun

On Mon, Dec 17, 2012 at 6:03 AM, 永辉 赵 <[EMAIL PROTECTED]> wrote:

> Hi all,
>
> In consumer we use a topic count map to set how many stream we want to
> split the topic.
> What's the relation between kafka partition number and consumer stream
> number?
> Seems  stream number should be less or equal to partition number?
> Because if we set stream number bigger than partition number then some
> stream is empty or some partition is split.
>
> Thanks,
> Yonghui
>
>
>
>