Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> consumer can't consume data any more when change the number of kafka partitions


Copy link to this message
-
Re: consumer can't consume data any more when change the number of kafka partitions
Have you set your consumer fetch.size larger than the max.messsage.size on
your producers?

Thanks,
Neha
On Jun 3, 2013 4:04 AM, "shangan chen" <[EMAIL PROTECTED]> wrote:

> This is a testing log in non-storm enviroment, it can't fetch any messages
> and seems to die there.
>
>
> 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: Connecting to
> zookeeper instance at 192.168.111.31:2181,192.168.111.40:2181,
> 192.168.111.41:2181,192.168.111.51:2181,192.168.111.62:2181
> 13/06/03 18:44:55 INFO zkclient.ZkEventThread: Starting ZkClient event
> thread.
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client environment:host.name
> =zjm_111_31
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0_25
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=Sun Microsystems Inc.
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/opt/apps_install/jdk-1.6.0_25/jre
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=log_storm_0.1-jar-with-dependencies.jar
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
>
> environment:java.library.path=/opt/apps_install/jdk-1.6.0_25/jre/lib/amd64/server:/opt/apps_install/jdk-1.6.0_25/jre/lib/amd64:/opt/apps_install/jdk-1.6.0_25/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client environment:os.name
> =Linux
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:os.version=2.6.18-308.el5xen
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client environment:user.name
> =root
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:user.home=/root
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:user.dir=/opt/services/storm
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=192.168.111.31:2181,192.168.111.40:2181,192.168.111.41:2181,
> 192.168.111.51:2181,192.168.111.62:2181 sessionTimeout=6000
> watcher=org.I0Itec.zkclient.ZkClient@575fadcf
> 13/06/03 18:44:55 INFO zookeeper.ClientCnxn: Opening socket connection to
> server zjm_111_62/192.168.111.62:2181. Will not attempt to authenticate
> using SASL (Unable to locate a login configuration)
> 13/06/03 18:44:55 INFO zookeeper.ClientCnxn: Socket connection established
> to zjm_111_62/192.168.111.62:2181, initiating session
> 13/06/03 18:44:55 INFO zookeeper.ClientCnxn: Session establishment complete
> on server zjm_111_62/192.168.111.62:2181, sessionid = 0x53e635c520c0140,
> negotiated timeout = 6000
> 13/06/03 18:44:55 INFO zkclient.ZkClient: zookeeper state changed
> (SyncConnected)
> 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: starting auto
> committer every 1000 ms
> 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: begin
> registering consumer login-group_zjm_111_31-1370256295616-2d287a98 in ZK
> 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: end registering
> consumer login-group_zjm_111_31-1370256295616-2d287a98 in ZK
> 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: begin
> rebalancing consumer login-group_zjm_111_31-1370256295616-2d287a98 try #0
> 13/06/03 18:44:56 INFO consumer.ZookeeperConsumerConnector: Committing all
> offsets
> 13/06/03 18:44:56 INFO consumer.ZookeeperConsumerConnector: Releasing
> partition ownership
> 13/06/03 18:44:56 INFO consumer.ZookeeperConsumerConnector: Consumer
> login-group_zjm_111_31-1370256295616-2d287a98 rebalancing the following
> partitions: List(1-0, 2-0) for topic login with consumers:
> List(login-group_zjm_109_152-1370253495249-eb0a173f-0,