Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Re: Socket timeouts in 0.8


+
Bob Jervis 2013-03-22, 16:38
+
Jun Rao 2013-03-22, 17:08
Copy link to this message
-
Re: Socket timeouts in 0.8
I am getting the logs and I am trying to make sense of them.  I see a
'Received Request' log entry that appears to be what is coming in from our
app.  I don't see any 'Completed Request' entries that correspond to those.
 The only completed entries I see for the logs in question are from the
replica-fetcher.

It is as if our app is asking the wrong broker and getting no answer, but
for some reason reporting it as a socket timeout.

Broker 0 is getting and completing TopicMetadata requests in about 600
milliseconds each.
Broker 1 is not reporting ANY TopicMetadatRequests in the TRACE logs.

Our app logs don't make any sense when I compare them to the broker logs
and how can we be getting timeouts in less than 1000 milliseconds?

Our app is reporting this:

2013-03-22 17:42:23,047 WARN kafka.producer.async.DefaultEventHandler:
failed to send to broker 1 with data Map([v1-english-5,0] ->
ByteBufferMessageSet(MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 2606857931, key = null, payload = java.nio.HeapByteBuffer[pos=0
lim=1700 cap=1700]),0), MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 735213417, key = null, payload = java.nio.HeapByteBuffer[pos=0
lim=1497 cap=1497]),1), MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 2435755724, key = null, payload = java.nio.HeapByteBuffer[pos=0
lim=1494 cap=1494]),2), MessageAndOffset(Message(magic = 0, attributes = 0,
crc = 202370440, key = null, paylo.....
java.net.SocketTimeoutException
        at
sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201)
        at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
        at
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:221)
        at kafka.utils.Utils$.read(Utils.scala:372)
        at
kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
        at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
        at
kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
        at kafka.network.BlockingChannel.receive(BlockingChannel.scala:100)
        at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
        at
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
        at
kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:98)
        at
kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:98)
        at
kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:98)
        at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
        at
kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:97)
        at
kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:97)
        at
kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:97)
        at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
        at kafka.producer.SyncProducer.send(SyncProducer.scala:96)
        at
kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:221)
        at
kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:91)
        at
kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:85)
        at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
        at
scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
        at scala.collection.Iterator$class.foreach(Iterator.scala:631)
        at
scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
        at
scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
        at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
        at
kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:85)
        at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:61)
        at kafka.producer.Producer.send(Producer.scala:76)
        at kafka.javaapi.producer.Producer.send(Producer.scala:41)
        at
com.visibletechnologies.platform.common.kafka.KafkaWriter.flush(KafkaWriter.java:131)
        at
com.visibletechnologies.platform.common.kafka.KafkaWriter.checkFlush(KafkaWriter.java:97)
        at
com.visibletechnologies.platform.katta.krouter.KRouter.checkFlush(KRouter.java:182)
        at
com.visibletechnologies.platform.katta.krouter.KRouter.doWork(KRouter.java:139)
        at
com.visibletechnologies.framework.servicebase.ServiceBase.start(ServiceBase.java:187)
        at
com.visibletechnologies.platform.katta.krouter.Main.main(Main.java:132)
2013-03-22 17:42:23,157 INFO kafka.client.ClientUtils$: Fetching metadata
for topic Set(v1-japanese-0, v1-indonesian-5, v1-french-3, v1-other-4,
v1-portuguese-1, v1-other-1, v1-german-2, v1-english-15, v1-english-8,
v1-portuguese-6, v1-arabic-0, v1-english-6, v1-spanish-5, v1-english-10,
v1-japanese-4, v1-english-1, v1-italian-3, v1-spanish-1, v1-english-5,
v1-other-7, v1-portuguese-2, v1-other-0, v1-indonesian-4, v1-english-9,
v1-japanese-1, v1-spanish-2, v1-portuguese-7, v1-german-3, v1-arabic-1,
v1-japanese-7, v1-korean-0, v1-spanish-4, v1-russian-0, v1-other-3,
v1-japanese-3, v1-english-0, v1-french-1, v1-indonesian-0, v1-arabic-2,
v1-english-4, v1-french-0, v1-indonesian-7, v1-english-12, v1-spanish-3,
v1-japanese-6, v1-indonesian-3, v1-english-13, v1-other-6, v1-portuguese-3,
v1-italian-1, v1-english-7, v1-german-0, v1-korean-3, v1-spanish-7,
v1-russian-1, v1-indonesian-1, v1-portuguese-0, v1-indonesian-6,
v1-french-2, v1-russian-2, v1-english-11, v1-japanese-2, v1-other-2,
v1-indonesian-2, v1-english-14, v1-portuguese-5, v1-spanish-6, v1-german-1,
v1-italian-0, v1-english-2, v1-japa
+
Bob Jervis 2013-03-22, 18:28
+
Neha Narkhede 2013-03-22, 19:54
+
Bob Jervis 2013-03-22, 23:25
+
Bob Jervis 2013-03-22, 16:44