Kafka, mail # user - Fwd: Issue in using kafka SimpleConsumer code - 2014-08-19, 18:48
Solr & Elasticsearch trainings in New York & San Francisco [more info][hide]
 Search Hadoop and all its subprojects:

Switch to Threaded View
Copy link to this message
-
Fwd: Issue in using kafka SimpleConsumer code
Hi Team,

I have a requirement of reading real time data using kafka and write to
cassandra.
For this I am using SimpleConsumer to read data from Kafka topics and
writing into Cassandra.
I am maintaining offsets of topics in my log files.
The issue is that after few days like 3-4 days my cosumer code does not
read data from kafka topics and produce below log output :

20:01:17,068  INFO NposKafkaConsumer:48 - Taking partition from application
properties
20:01:17,482 DEBUG BlockingChannel:52 - Created socket with SO_TIMEOUT =
100000 (requested 100000), SO_RCVBUF = 65536 (requested 65536), SO_SNDBUF =
64512 (requested -1).
20:01:17,545 DEBUG SimpleConsumer:52 - Disconnecting from <IP address>:9092
20:01:17,578 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method
Inside while loop :: Value of max_reads::1
20:01:17,662 DEBUG BlockingChannel:52 - Created socket with SO_TIMEOUT =
100000 (requested 100000), SO_RCVBUF = 65536 (requested 65536), SO_SNDBUF =
64512 (requested -1).
20:01:17,804 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping
20:01:18,804 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method
Inside while loop :: Value of max_reads::1
20:01:18,826 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping
20:01:19,827 DEBUG NposKafkaConsumer:113 - NposKafkaConsumer.run() method
Inside while loop :: Value of max_reads::1
20:01:19,852 DEBUG NposKafkaConsumer:193 - numRead::0 Sleeping

NposKafkaConsumer is my main SimpleCosumer class.
But, when I restart the kafka process by incrementing the offset by one
then again my code starts running fine for next few days.

Can you please help me how I can solve this and where I am going wrong ?

Thanks & Regards,
Pankaj Ojha

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB