Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Socket timeouts in 0.8


Copy link to this message
-
Re: Socket timeouts in 0.8
I am getting the logs and I am trying to make sense of them.  I see a
'Received Request' log entry that appears to be what is coming in from our
app.  I don't see any 'Completed Request' entries that correspond to those.
 The only completed entries I see for the logs in question are from the
replica-fetcher.

It is as if our app is asking the wrong broker and getting no answer, but
for some reason reporting it as a socket timeout.

Broker 0 is getting and completing TopicMetadata requests in about 600
milliseconds each.
Broker 1 is not reporting ANY TopicMetadatRequests in the TRACE logs.

Our app logs don't make any sense when I compare them to the broker logs
and how can we be getting timeouts in less than 1000 milliseconds?

Our app is reporting this:

2013-03-22 17:42:23,047 WARN kafka.producer.async.DefaultEventHandler:
failed to send to broker 1 with data Map([v1-english-5,0] ->
ByteBufferMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 2606857931, key = null, payload = java.nio.HeapByteBuffer[pos=0
lim=1700 cap=1700]),0), MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 735213417, key = null, payload = java.nio.HeapByteBuffer[pos=0
lim=1497 cap=1497]),1), MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 2435755724, key = null, payload = java.nio.HeapByteBuffer[pos=0
lim=1494 cap=1494]),2), MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 202370440, key = null, paylo.....
java.net.SocketTimeoutException
        at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
        at
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:221)
        at kafka.utils.Utils$.read(Utils.scala:372)
        at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
        at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
        at
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
        at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
        at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
        at
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
        at
kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:98)
        at
kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:98)
        at
kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:98)
        at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
        at
kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:97)
        at
kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:97)
        at
kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:97)
        at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:96)
        at
kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:221)
        at
kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:91)
        at
kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:85)
        at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
        at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
        at scala.collection.Iterator$class.foreach(Iterator.scala:631)
        at
scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
        at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
        at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
        at
kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:85)
        at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:61)
        at kafka.producer.Producer.send(Producer.scala:76)
        at kafka.javaapi.producer.Producer.send(Producer.scala:41)
        at
com.visibletechnologies.platform.common.kafka.KafkaWriter.flush(KafkaWriter.java:131)
        at
com.visibletechnologies.platform.common.kafka.KafkaWriter.checkFlush(KafkaWriter.java:97)
        at
com.visibletechnologies.platform.katta.krouter.KRouter.checkFlush(KRouter.java:182)
        at
com.visibletechnologies.platform.katta.krouter.KRouter.doWork(KRouter.java:139)
        at
com.visibletechnologies.framework.servicebase.ServiceBase.start(ServiceBase.java:187)
        at
com.visibletechnologies.platform.katta.krouter.Main.main(Main.java:132)
2013-03-22 17:42:23,157 INFO kafka.client.ClientUtils$: Fetching metadata
for topic Set(v1-japanese-0, v1-indonesian-5, v1-french-3, v1-other-4,
v1-portuguese-1, v1-other-1, v1-german-2, v1-english-15, v1-english-8,
v1-portuguese-6, v1-arabic-0, v1-english-6, v1-spanish-5, v1-english-10,
v1-japanese-4, v1-english-1, v1-italian-3, v1-spanish-1, v1-english-5,
v1-other-7, v1-portuguese-2, v1-other-0, v1-indonesian-4, v1-english-9,
v1-japanese-1, v1-spanish-2, v1-portuguese-7, v1-german-3, v1-arabic-1,
v1-japanese-7, v1-korean-0, v1-spanish-4, v1-russian-0, v1-other-3,
v1-japanese-3, v1-english-0, v1-french-1, v1-indonesian-0, v1-arabic-2,
v1-english-4, v1-french-0, v1-indonesian-7, v1-english-12, v1-spanish-3,
v1-japanese-6, v1-indonesian-3, v1-english-13, v1-other-6, v1-portuguese-3,
v1-italian-1, v1-english-7, v1-german-0, v1-korean-3, v1-spanish-7,
v1-russian-1, v1-indonesian-1, v1-portuguese-0, v1-indonesian-6,
v1-french-2, v1-russian-2, v1-english-11, v1-japanese-2, v1-other-2,
v1-indonesian-2, v1-english-14, v1-portuguese-5, v1-spanish-6, v1-german-1,
v1-italian-0, v1-english-2, v1-japa
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB