Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Getting timeouts with elastic load balancer in AWS


Copy link to this message
-
Getting timeouts with elastic load balancer in AWS
Hi all,

We are sending our ad impressions to Kafka 0.7.0. I am using async
prouducers in our web app.
I am pooling kafak producers with commons pool. Pool size - 10. batch.size
is 100.

We have 3 c1.xlarge instances with Kafka brokers installed behind a elastic
load balancer in AWS.
Every minute we loose some events because of the following exception

- Disconnecting from dualstack.kafka-xyz.us-east-1.elb.amazonaws.com:9092
- Error in handling batch of 64 events
java.io.IOException: Connection timed out
    at sun.nio.ch.FileDispatcher.write0(Native Method)
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
    at sun.nio.ch.IOUtil.write(IOUtil.java:75)
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
    at
kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:51)
    at kafka.network.Send$class.writeCompletely(Transmission.scala:76)
    at
kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
    at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:88)
    at kafka.producer.SyncProducer.send(SyncProducer.scala:87)
    at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:128)
    at
kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:52)
    at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:46)
    at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:119)
    at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:98)
    at
kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:74)
    at scala.collection.immutable.Stream.foreach(Stream.scala:254)
    at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:73)
    at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:43)
- Connected to dualstack.kafka-xyz.us-east-1.elb.amazonaws.com:9092 for
producing

Has anybody faced this kind of timeouts before? Do they indicate any
resource misconfiguration? The CPU usage on broker is pretty small.
Also, in spite of setting batch size to 100, the failing batch usually only
have 50 to 60 events. Is there any other limit I am hitting?

Any help is appreciated.
Regards,
Vaibhav
GumGum