Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> consumer can't consume data any more when change the number of kafka partitions


Copy link to this message
-
Re: consumer can't consume data any more when change the number of kafka partitions
Have you followed #6 and #7 in http://kafka.apache.org/faq.html?

Thanks,

Jun
On Tue, Jun 4, 2013 at 10:04 PM, shangan chen <[EMAIL PROTECTED]>wrote:

> I do have more consumers than partitions, but I don't think this is the
> reason, as all consumers seem to die, no messages go to any consumer.
> I stop other consumers, and the one that own a partition, but still can't
> fetch data.
>
> 13/06/03 18:49:16 INFO consumer.ZookeeperConsumerConnector: Consumer
> login-group_zjm_111_31-1370256295616-2d287a98 selected partitions :
> 13/06/03 18:49:16 INFO consumer.ZookeeperConsumerConnector: end rebalancing
> consumer login-group_zjm_111_31-1370256295616-2d287a98 try #0
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector: begin
> rebalancing consumer login-group_zjm_111_31-1370256295616-2d287a98 try #0
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector: Committing all
> offsets
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector: Releasing
> partition ownership
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector: Consumer
> login-group_zjm_111_31-1370256295616-2d287a98 rebalancing the following
> partitions: List(1-0, 2-0) for topic login with consumers:
> List(login-group_zjm_111_31-1370256295616-2d287a98-0,
> login-group_zjm_111_31-1370256295616-2d287a98-1)
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector:
> login-group_zjm_111_31-1370256295616-2d287a98-1 attempting to claim
> partition 2-0
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector:
> login-group_zjm_111_31-1370256295616-2d287a98-0 attempting to claim
> partition 1-0
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector: Consumer
> login-group_zjm_111_31-1370256295616-2d287a98 selected partitions :
> login:1-0: fetched offset = 482098298: consumed offset =
> 482098298,login:2-0: fetched offset = 385686015: consumed offset =
> 385686015
> 13/06/03 18:49:18 INFO consumer.FetcherRunnable: FetchRunnable-0 start
> fetching topic: login part: 0 offset: 482098298 from 192.168.111.40:9092
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector: end rebalancing
> consumer login-group_zjm_111_31-1370256295616-2d287a98 try #0
> 13/06/03 18:49:18 INFO consumer.FetcherRunnable: FetchRunnable-1 start
> fetching topic: login part: 0 offset: 385686015 from 192.168.111.51:9092
>
>
> On Mon, Jun 3, 2013 at 11:24 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > From the following log, it seems that you have more consumers than
> > partitions. So, some consumers will never get any data. Please see #4 in
> > http://kafka.apache.org/faq.html for details.
> >
> > 13/06/03 18:49:16 INFO consumer.ZookeeperConsumerConnector: Consumer
> > login-group_zjm_111_31-1370256295616-2d287a98 rebalancing the following
> > partitions: List(1-0, 2-0) for topic login with consumers:
> > List(login-group_zjm_109_152-1370253495249-eb0a173f-0,
> > login-group_zjm_109_152-1370253495249-eb0a173f-1,
> > login-group_zjm_110_159-1370253496049-119021ef-0,
> > login-group_zjm_110_159-1370253496049-119021ef-1,
> > login-group_zjm_111_31-1370256295616-2d287a98-0,
> > login-group_zjm_111_31-1370256295616-2d287a98-1,
> > login-group_zjm_111_49-1370253495793-26c03d19-0,
> > login-group_zjm_111_49-1370253495793-26c03d19-1,
> > login-group_zjm_111_49-1370253496335-c9be525d-0,
> > login-group_zjm_111_49-1370253496335-c9be525d-1)
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Jun 3, 2013 at 4:04 AM, shangan chen <[EMAIL PROTECTED]
> > >wrote:
> >
> > > This is a testing log in non-storm enviroment, it can't fetch any
> > messages
> > > and seems to die there.
> > >
> > >
> > > 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: Connecting
> to
> > > zookeeper instance at 192.168.111.31:2181,192.168.111.40:2181,
> > > 192.168.111.41:2181,192.168.111.51:2181,192.168.111.62:2181
> > > 13/06/03 18:44:55 INFO zkclient.ZkEventThread: Starting ZkClient event
> > > thread.
> > > 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> > > environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB