Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Fwd: Issue in using kafka SimpleConsumer code


Copy link to this message
-
Fwd: Issue in using kafka SimpleConsumer code
Hi Team,

I have a requirement of reading real time data using kafka and write to
cassandra.
For this I am using SimpleConsumer to read data from Kafka topics and
writing into Cassandra.
I am maintaining offsets of topics in my log files.
The issue is that after few days like 3-4 days my cosumer code does not
read data from kafka topics and produce below log output :

20:01:17,068  INFO NposKafkaConsumer:48 - Taking partition from application
properties
20:01:17,482 DEBUG BlockingChannel:52 - Created socket with SO_TIMEOUT =
100000 (requested 100000), SO_RCVBUF = 65536 (requested 65536), SO_SNDBUF =
64512 (requested -1).
20:01:17,545 DEBUG SimpleConsumer:52 - Disconnecting from <IP address>:9092
20:01:17,578 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method
Inside while loop :: Value of max_reads::1
20:01:17,662 DEBUG BlockingChannel:52 - Created socket with SO_TIMEOUT =
100000 (requested 100000), SO_RCVBUF = 65536 (requested 65536), SO_SNDBUF =
64512 (requested -1).
20:01:17,804 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping
20:01:18,804 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method
Inside while loop :: Value of max_reads::1
20:01:18,826 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping
20:01:19,827 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method
Inside while loop :: Value of max_reads::1
20:01:19,852 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping

NposKafkaConsumer is my main SimpleCosumer class.
But, when I restart the kafka process by incrementing the offset by one
then again my code starts running fine for next few days.

Can you please help me how I can solve this and where I am going wrong ?

Thanks & Regards,
Pankaj Ojha

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB