Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Consumer in the mirror maker stop fetching logs


Copy link to this message
-
Consumer in the mirror maker stop fetching logs
hi,

We are doing some mirror-maker test。We currently use version 0.7.2。

some times,the mirror maker will throw exceptions like this, and then
stopped fetching topics:

java.io.EOFException: Received -1 when reading from channel, socket has
likely been closed.
     at kafka.utils.Utils$.read(Utils.scala:539)
     at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67)
     at kafka.network.Receive$class.readCompletely(Transmission.scala:55)
     at
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
     at kafka.consumer.SimpleConsumer.getResponse(SimpleConsumer.scala:177)
     at
kafka.consumer.SimpleConsumer.liftedTree2$1(SimpleConsumer.scala:125)
     at kafka.consumer.SimpleConsumer.multifetch(SimpleConsumer.scala:115)
     at kafka.consumer.FetcherRunnable.run(FetcherRunnable.scala:60)
[2013-05-02 15:13:05,339] INFO stopping fetcher FetchRunnable-0 to host
kf.a.com (kafka.consumer.FetcherRunnable)

I used jconsole to double check, and confirmed that the consumer in the
mirror maker stopped fetching logs brokers。I have to stop the mirror maker
and start it over, and it continued fetching logs。

I start consumers using only 1 consumer thread。

By the way,there are 16 mirror makers in the consume group。
I did some calculation and it can be assured that the partition numbers are
larger than or equal to the consumer threads。

Thats all I got。So, I wonder if there was a bug issue for just mirror maker
or any applications using the high level consumers will behave like this?
thanks!
--
*Best Regards

Helin Xiang*

 
+
Jun Rao 2013-05-03, 15:50
+
Helin Xiang 2013-05-04, 11:58
+
Jun Rao 2013-05-06, 01:05
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB