Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> Occasional batch send errors


Copy link to this message
-
Re: Occasional batch send errors
Usually when these types of errors are because you're not connecting to the
proper host:port. Double check your configs, make sure everything is
running and listening on the host:port you think they are.

Have you tried using the sync producer to work out your bugs? My guess is
the sync producer would fail on the first message rather than failing when
the batch is submitted.
On Tue, Apr 23, 2013 at 4:57 PM, Karl Kirch <[EMAIL PROTECTED]> wrote:

> Hmmm… that didn't seem to help.
> Anyone else see this sort of errors?
>
> Karl
>
>
> On Apr 23, 2013, at 5:58 PM, Karl Kirch <[EMAIL PROTECTED]>
>  wrote:
>
> > I'm going to try bumping up the "numRetries" key in my producer config.
> > Is this a good option in this case?
> > I am using the zookeeper connect option so I'm aware that I may get
> stuck retrying to a failed node, but if it's just a temporary network
> glitch I'll at least get a bit more of a chance to recover.
> >
> > Thanks,
> > Karl
> >
> > On Apr 23, 2013, at 5:35 PM, Karl Kirch <[EMAIL PROTECTED]>
> > wrote:
> >
> >> I occasionally am getting some batch send errors from the stock async
> producer. This is on a cluster of 3 kafka (0.7.2) and 3 zookeeper nodes.
> >> Is there anyway to check what happens when those batch errors occur?
> >> Or bump up the retry count? (looks like it only did a single retry).
> >>
> >> I need the speed of the async producer, but it needs to be reliable (I
> see a handful of these a day but in a weather alerting system it only takes
> missing one let alone 25 or 100/1000).
> >>
> >> Here's a stack trace of one of the errors that I'm seeing.
> >>
> >> 22:23:39.405 [ProducerSendThread-1824508747] WARN
>  k.p.a.DefaultEventHandler - Error sending messages, 0 attempts remaining
> >> java.io.IOException: Connection reset by peer
> >>      at sun.nio.ch.FileDispatcher.writev0(Native Method) ~[na:1.6.0_24]
> >>      at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:33)
> ~[na:1.6.0_24]
> >>      at sun.nio.ch.IOUtil.write(IOUtil.java:125) ~[na:1.6.0_24]
> >>      at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:367)
> ~[na:1.6.0_24]
> >>      at java.nio.channels.SocketChannel.write(SocketChannel.java:360)
> ~[na:1.6.0_24]
> >>      at
> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:49)
> ~[apns-consumer-1.0.jar:na]
> >>      at kafka.network.Send$class.writeCompletely(Transmission.scala:73)
> ~[apns-consumer-1.0.jar:na]
> >>      at
> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:25)
> ~[apns-consumer-1.0.jar:na]
> >>      at
> kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:95)
> ~[apns-consumer-1.0.jar:na]
> >>      at kafka.producer.SyncProducer.send(SyncProducer.scala:94)
> ~[apns-consumer-1.0.jar:na]
> >>      at kafka.producer.SyncProducer.multiSend(SyncProducer.scala:135)
> ~[apns-consumer-1.0.jar:na]
> >>      at
> kafka.producer.async.DefaultEventHandler.send(DefaultEventHandler.scala:58)
> [apns-consumer-1.0.jar:na]
> >>      at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:44)
> [apns-consumer-1.0.jar:na]
> >>      at
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:116)
> [apns-consumer-1.0.jar:na]
> >>      at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:95)
> [apns-consumer-1.0.jar:na]
> >>      at
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:71)
> [apns-consumer-1.0.jar:na]
> >>      at scala.collection.immutable.Stream.foreach(Stream.scala:260)
> [apns-consumer-1.0.jar:na]
> >>      at
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:70)
> [apns-consumer-1.0.jar:na]
> >>      at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:41)
> [apns-consumer-1.0.jar:na]
> >> 22:23:39.406 [ProducerSendThread-1824508747] ERROR
> k.p.a.ProducerSendThread - Error in handling batch of 27 events

 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB