Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> consumer can't consume data any more when change the number of kafka partitions


Copy link to this message
-
Re: consumer can't consume data any more when change the number of kafka partitions
From the following log, it seems that you have more consumers than
partitions. So, some consumers will never get any data. Please see #4 in
http://kafka.apache.org/faq.html for details.

13/06/03 18:49:16 INFO consumer.ZookeeperConsumerConnector: Consumer
login-group_zjm_111_31-1370256295616-2d287a98 rebalancing the following
partitions: List(1-0, 2-0) for topic login with consumers:
List(login-group_zjm_109_152-1370253495249-eb0a173f-0,
login-group_zjm_109_152-1370253495249-eb0a173f-1,
login-group_zjm_110_159-1370253496049-119021ef-0,
login-group_zjm_110_159-1370253496049-119021ef-1,
login-group_zjm_111_31-1370256295616-2d287a98-0,
login-group_zjm_111_31-1370256295616-2d287a98-1,
login-group_zjm_111_49-1370253495793-26c03d19-0,
login-group_zjm_111_49-1370253495793-26c03d19-1,
login-group_zjm_111_49-1370253496335-c9be525d-0,
login-group_zjm_111_49-1370253496335-c9be525d-1)

Thanks,

Jun
On Mon, Jun 3, 2013 at 4:04 AM, shangan chen <[EMAIL PROTECTED]>wrote:

> This is a testing log in non-storm enviroment, it can't fetch any messages
> and seems to die there.
>
>
> 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: Connecting to
> zookeeper instance at 192.168.111.31:2181,192.168.111.40:2181,
> 192.168.111.41:2181,192.168.111.51:2181,192.168.111.62:2181
> 13/06/03 18:44:55 INFO zkclient.ZkEventThread: Starting ZkClient event
> thread.
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client environment:host.name
> =zjm_111_31
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.version=1.6.0_25
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.vendor=Sun Microsystems Inc.
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.home=/opt/apps_install/jdk-1.6.0_25/jre
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.class.path=log_storm_0.1-jar-with-dependencies.jar
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
>
> environment:java.library.path=/opt/apps_install/jdk-1.6.0_25/jre/lib/amd64/server:/opt/apps_install/jdk-1.6.0_25/jre/lib/amd64:/opt/apps_install/jdk-1.6.0_25/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.io.tmpdir=/tmp
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:java.compiler=<NA>
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client environment:os.name
> =Linux
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:os.arch=amd64
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:os.version=2.6.18-308.el5xen
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client environment:user.name
> =root
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:user.home=/root
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Client
> environment:user.dir=/opt/services/storm
> 13/06/03 18:44:55 INFO zookeeper.ZooKeeper: Initiating client connection,
> connectString=192.168.111.31:2181,192.168.111.40:2181,192.168.111.41:2181,
> 192.168.111.51:2181,192.168.111.62:2181 sessionTimeout=6000
> watcher=org.I0Itec.zkclient.ZkClient@575fadcf
> 13/06/03 18:44:55 INFO zookeeper.ClientCnxn: Opening socket connection to
> server zjm_111_62/192.168.111.62:2181. Will not attempt to authenticate
> using SASL (Unable to locate a login configuration)
> 13/06/03 18:44:55 INFO zookeeper.ClientCnxn: Socket connection established
> to zjm_111_62/192.168.111.62:2181, initiating session
> 13/06/03 18:44:55 INFO zookeeper.ClientCnxn: Session establishment complete
> on server zjm_111_62/192.168.111.62:2181, sessionid = 0x53e635c520c0140,
> negotiated timeout = 6000
> 13/06/03 18:44:55 INFO zkclient.ZkClient: zookeeper state changed
> (SyncConnected)
> 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: starting auto
> committer every 1000 ms
> 13/06/03 18:44:55 INFO consumer.ZookeeperConsumerConnector: begin

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB