Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - Consumer group questions


Copy link to this message
-
Re: Consumer group questions
Mark 2013-10-09, 23:03
> Will the topics be distributed across both machines or will it still be all
> read from the first process that spawned up the 9 threads?
>
> It will read from the first process that spawned 9 threads.

I should have actually asked the opposite of this. Say I have a consumer running 3 threads on 3 different machines. If one machine crashes is there anything I can do to make sure those 3 partitions are still read from?

On Oct 9, 2013, at 1:18 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote:

> Will the topics be distributed across both machines or will it still be all
> read from the first process that spawned up the 9 threads?
>
> It will read from the first process that spawned 9 threads.
>
> In general, is it better to have 1 machine running 9 threads to read all
> partitions or 9 machines running 1 thread?
>
> It really depends on the network bandwidth and CPU usage of your
> application.
>
> I've been trying to work with (
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example)
> and it appears to be for 0.8
>
> The code is not different from the high level consumer in 0.7. You can see
> the 0.7 configs here - http://kafka.apache.org/07/configuration.html
>
> Thanks,
> Neha
>
>
> On Wed, Oct 9, 2013 at 10:07 AM, Mark <[EMAIL PROTECTED]> wrote:
>
>> This is in regards to consumer group consumption in 0.7.2.
>>
>> Say we have 3 machines with 3 partitions in each topic totaling 9
>> partitions. Now if I create a consumer group with 9 threads on the same
>> machine then all partitions will be read from. Now what happens if I start
>> another 9 threads on a separate machine using the same consume group id?
>> Will the topics be distributed across both machines or will it still be all
>> read from the first process that spawned up the 9 threads?
>>
>> In general, is it better to have 1 machine running 9 threads to read all
>> partitions or 9 machines running 1 thread?
>>
>> Is there a wiki for 0.7.2? I've been trying to work with (
>> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example)
>> and it appears to be for 0.8
>>
>> Thanks