I have a problem in fetching messages from Kafka. I am using simple consumer API in Java to fetch messages from kafka ( the same one which is stated in Kafka introduction example). The problem is that after a while (could be 30min or couple of hours), the consumer does not receive any messages from Kafka, while the data exist there (while the streaming of data to Kafka still running, so Kafka has inputs). I can see that data exist in Kafka by just running the following command and getting the list of messages exist in Kafka, Each message is around 80 bytes :
*bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test Any idea what could be the source of problem?? I also notices that if i stress the input to kafka (sending 1000 messages per scond) for an hour or more , the same situation happens again. ?? It seems that something is wrong with fetching (consumer) part, right?
The kafka is run in one machine, no clusters, replications,....etc, very basic configuration.
The consumer config file is ;
"zookeeper.connect", myserver:2181); "group.id", group1); "zookeeper.session.timeout.ms", "400"); "zookeeper.sync.time.ms", "200"); "auto.commit.interval.ms", "1000"); "fetch.message.max.bytes", "1048576"); "auto.offset.reset", "smallest"; and the server.config looks like this:
Thanks a lot, very good hints. I am trying to see what happened in my case.
best, /Shahab On Wed, Dec 11, 2013 at 5:16 PM, Jun Rao <[EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext