Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Consumer group questions


Copy link to this message
-
Re: Consumer group questions
> Will the topics be distributed across both machines or will it still be all
> read from the first process that spawned up the 9 threads?
>
> It will read from the first process that spawned 9 threads.

I should have actually asked the opposite of this. Say I have a consumer running 3 threads on 3 different machines. If one machine crashes is there anything I can do to make sure those 3 partitions are still read from?

On Oct 9, 2013, at 1:18 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote:

> Will the topics be distributed across both machines or will it still be all
> read from the first process that spawned up the 9 threads?
>
> It will read from the first process that spawned 9 threads.
>
> In general, is it better to have 1 machine running 9 threads to read all
> partitions or 9 machines running 1 thread?
>
> It really depends on the network bandwidth and CPU usage of your
> application.
>
> I've been trying to work with (
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example)
> and it appears to be for 0.8
>
> The code is not different from the high level consumer in 0.7. You can see
> the 0.7 configs here - http://kafka.apache.org/07/configuration.html
>
> Thanks,
> Neha
>
>
> On Wed, Oct 9, 2013 at 10:07 AM, Mark <[EMAIL PROTECTED]> wrote:
>
>> This is in regards to consumer group consumption in 0.7.2.
>>
>> Say we have 3 machines with 3 partitions in each topic totaling 9
>> partitions. Now if I create a consumer group with 9 threads on the same
>> machine then all partitions will be read from. Now what happens if I start
>> another 9 threads on a separate machine using the same consume group id?
>> Will the topics be distributed across both machines or will it still be all
>> read from the first process that spawned up the 9 threads?
>>
>> In general, is it better to have 1 machine running 9 threads to read all
>> partitions or 9 machines running 1 thread?
>>
>> Is there a wiki for 0.7.2? I've been trying to work with (
>> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example)
>> and it appears to be for 0.8
>>
>> Thanks
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB