Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka, mail # user - consumer can't consume data any more when change the number of kafka partitions


Copy link to this message
-
Re: consumer can't consume data any more when change the number of kafka partitions
Neha Narkhede 2013-06-03, 15:01
Have you set your consumer fetch.size larger than the max.messsage.size on
your producers?

Thanks,
Neha
On Jun 3, 2013 4:04 AM, "shangan chen" <[EMAIL PROTECTED]> wrote:

> This is a testing log in non-storm enviroment, it can't fetch any messages
> and seems to die there.
>
>
> 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: Connecting to
> zookeeper instance at 192.168.111.31:2181,192.168.111.40:2181,
> 192.168.111.41:2181,192.168.111.51:2181,192.168.111.62:2181
> 13/06/03 18:44:55 INFO zkclient.ZkEventThread: Starting ZkClient event
> thread.
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client environment:host.name
> =zjm_111_31
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0_25
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=Sun Microsystems Inc.
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/opt/apps_install/jdk-1.6.0_25/jre
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=log_storm_0.1-jar-with-dependencies.jar
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
>
> environment:java.library.path=/opt/apps_install/jdk-1.6.0_25/jre/lib/amd64/server:/opt/apps_install/jdk-1.6.0_25/jre/lib/amd64:/opt/apps_install/jdk-1.6.0_25/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client environment:os.name
> =Linux
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:os.version=2.6.18-308.el5xen
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client environment:user.name
> =root
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:user.home=/root
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:user.dir=/opt/services/storm
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=192.168.111.31:2181,192.168.111.40:2181,192.168.111.41:2181,
> 192.168.111.51:2181,192.168.111.62:2181 sessionTimeout=6000
> watcher=org.I0Itec.zkclient.ZkClient@575fadcf
> 13/06/03 18:44:55 INFO zookeeper.ClientCnxn: Opening socket connection to
> server zjm_111_62/192.168.111.62:2181. Will not attempt to authenticate
> using SASL (Unable to locate a login configuration)
> 13/06/03 18:44:55 INFO zookeeper.ClientCnxn: Socket connection established
> to zjm_111_62/192.168.111.62:2181, initiating session
> 13/06/03 18:44:55 INFO zookeeper.ClientCnxn: Session establishment complete
> on server zjm_111_62/192.168.111.62:2181, sessionid = 0x53e635c520c0140,
> negotiated timeout = 6000
> 13/06/03 18:44:55 INFO zkclient.ZkClient: zookeeper state changed
> (SyncConnected)
> 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: starting auto
> committer every 1000 ms
> 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: begin
> registering consumer login-group_zjm_111_31-1370256295616-2d287a98 in ZK
> 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: end registering
> consumer login-group_zjm_111_31-1370256295616-2d287a98 in ZK
> 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: begin
> rebalancing consumer login-group_zjm_111_31-1370256295616-2d287a98 try #0
> 13/06/03 18:44:56 INFO consumer.ZookeeperConsumerConnector: Committing all
> offsets
> 13/06/03 18:44:56 INFO consumer.ZookeeperConsumerConnector: Releasing
> partition ownership
> 13/06/03 18:44:56 INFO consumer.ZookeeperConsumerConnector: Consumer
> login-group_zjm_111_31-1370256295616-2d287a98 rebalancing the following
> partitions: List(1-0, 2-0) for topic login with consumers:
> List(login-group_zjm_109_152-1370253495249-eb0a173f-0,