Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - consumer can't consume data any more when change the number of kafka partitions


Copy link to this message
-
Re: consumer can't consume data any more when change the number of kafka partitions
Jun Rao 2013-06-05, 14:36
Have you followed #6 and #7 in http://kafka.apache.org/faq.html?

Thanks,

Jun
On Tue, Jun 4, 2013 at 10:04 PM, shangan chen <[EMAIL PROTECTED]>wrote:

> I do have more consumers than partitions, but I don't think this is the
> reason, as all consumers seem to die, no messages go to any consumer.
> I stop other consumers, and the one that own a partition, but still can't
> fetch data.
>
> 13/06/03 18:49:16 INFO consumer.ZookeeperConsumerConnector: Consumer
> login-group_zjm_111_31-1370256295616-2d287a98 selected partitions :
> 13/06/03 18:49:16 INFO consumer.ZookeeperConsumerConnector: end rebalancing
> consumer login-group_zjm_111_31-1370256295616-2d287a98 try #0
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector: begin
> rebalancing consumer login-group_zjm_111_31-1370256295616-2d287a98 try #0
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector: Committing all
> offsets
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector: Releasing
> partition ownership
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector: Consumer
> login-group_zjm_111_31-1370256295616-2d287a98 rebalancing the following
> partitions: List(1-0, 2-0) for topic login with consumers:
> List(login-group_zjm_111_31-1370256295616-2d287a98-0,
> login-group_zjm_111_31-1370256295616-2d287a98-1)
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector:
> login-group_zjm_111_31-1370256295616-2d287a98-1 attempting to claim
> partition 2-0
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector:
> login-group_zjm_111_31-1370256295616-2d287a98-0 attempting to claim
> partition 1-0
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector: Consumer
> login-group_zjm_111_31-1370256295616-2d287a98 selected partitions :
> login:1-0: fetched offset = 482098298: consumed offset =
> 482098298,login:2-0: fetched offset = 385686015: consumed offset =
> 385686015
> 13/06/03 18:49:18 INFO consumer.FetcherRunnable: FetchRunnable-0 start
> fetching topic: login part: 0 offset: 482098298 from 192.168.111.40:9092
> 13/06/03 18:49:18 INFO consumer.ZookeeperConsumerConnector: end rebalancing
> consumer login-group_zjm_111_31-1370256295616-2d287a98 try #0
> 13/06/03 18:49:18 INFO consumer.FetcherRunnable: FetchRunnable-1 start
> fetching topic: login part: 0 offset: 385686015 from 192.168.111.51:9092
>
>
> On Mon, Jun 3, 2013 at 11:24 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
>
> > From the following log, it seems that you have more consumers than
> > partitions. So, some consumers will never get any data. Please see #4 in
> > http://kafka.apache.org/faq.html for details.
> >
> > 13/06/03 18:49:16 INFO consumer.ZookeeperConsumerConnector: Consumer
> > login-group_zjm_111_31-1370256295616-2d287a98 rebalancing the following
> > partitions: List(1-0, 2-0) for topic login with consumers:
> > List(login-group_zjm_109_152-1370253495249-eb0a173f-0,
> > login-group_zjm_109_152-1370253495249-eb0a173f-1,
> > login-group_zjm_110_159-1370253496049-119021ef-0,
> > login-group_zjm_110_159-1370253496049-119021ef-1,
> > login-group_zjm_111_31-1370256295616-2d287a98-0,
> > login-group_zjm_111_31-1370256295616-2d287a98-1,
> > login-group_zjm_111_49-1370253495793-26c03d19-0,
> > login-group_zjm_111_49-1370253495793-26c03d19-1,
> > login-group_zjm_111_49-1370253496335-c9be525d-0,
> > login-group_zjm_111_49-1370253496335-c9be525d-1)
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Mon, Jun 3, 2013 at 4:04 AM, shangan chen <[EMAIL PROTECTED]
> > >wrote:
> >
> > > This is a testing log in non-storm enviroment, it can't fetch any
> > messages
> > > and seems to die there.
> > >
> > >
> > > 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: Connecting
> to
> > > zookeeper instance at 192.168.111.31:2181,192.168.111.40:2181,
> > > 192.168.111.41:2181,192.168.111.51:2181,192.168.111.62:2181
> > > 13/06/03 18:44:55 INFO zkclient.ZkEventThread: Starting ZkClient event
> > > thread.
> > > 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> > > environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52