Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Netty/Avro IPC problem: channel closed


Copy link to this message
-
Re: Netty/Avro IPC problem: channel closed
Hi Yang,

>From the stack trace you posted it appears that you are using a version of
Avro prior to 1.5.2.  Which version are you using?  There have been a number
of significant changes recently to the RPC framework and the Netty
implementation in particular.  Could you please try to reproduce the problem
using Avro 1.5.2 or newer?  The problem may be resolved with an upgrade.  If
the problem still exists in the newer versions, it will be a lot easier to
diagnose/fix it if we can see stack traces from a post-1.5.2 version.

Thanks,
James
On Wed, Sep 14, 2011 at 1:08 PM, Yang <[EMAIL PROTECTED]> wrote:

> I'm always seeing these "channel closed " exceptions , with low
> probability, i.e. about every 10 hours under heavy load.
>
> I'm not sure if it's the server that got the channel closed or the
> client, so I included the exception stack from both sides.
> anybody has an idea how to debug this?
>
> also, let's say it does have a valid reason for closing this, what is
> my strategy of coping with this? I originally have many
> senders, due to the channel close exception, many of them died, after
> this, only 2 application threads remain, but they
> all seem blocked on trying to grab a connection from Netty's pool, so
> even if I create new sender threads, it seems they would still
> block. so how can I tell netty to "reset/replenish " its connections?
>
>
> Thanks a lot
> Yang
>
>
> client side:
>
>
>
>  WARN 16:51:02,079 Unexpected exception from downstream.
> java.nio.channels.ClosedChannelException
>        at
> org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:636)
>        at
> org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:369)
>        at
> org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
>        at org.jboss.netty.channel.Channels.write(Channels.java:632)
>        at
> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
>        at org.jboss.netty.channel.Channels.write(Channels.java:611)
>        at org.jboss.netty.channel.Channels.write(Channels.java:578)
>        at
> org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259)
>        at
> org.apache.avro.ipc.NettyTransceiver.transceive(NettyTransceiver.java:131)
>        at org.apache.avro.ipc.Requestor.request(Requestor.java:134)
>        at
> org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:55)
>        at $Proxy0.collect_ad_click(Unknown Source)
>
>
>
> server side:
>
>
>  WARN 16:51:01,939 Unexpected exception from downstream.
> java.io.IOException: Broken pipe
>        at sun.nio.ch.FileDispatcher.write0(Native Method)
>        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
>        at sun.nio.ch.IOUtil.write(IOUtil.java:78)
>        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)
>        at
> org.jboss.netty.channel.socket.nio.SocketSendBufferPool$PooledSendBuffer.transferTo(SocketSendBufferPool.java:239)
>        at
> org.jboss.netty.channel.socket.nio.NioWorker.write0(NioWorker.java:469)
>        at
> org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:387)
>        at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
>        at
> org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
>        at org.jboss.netty.channel.Channels.write(Channels.java:632)
>        at
> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
>        at org.jboss.netty.channel.Channels.write(Channels.java:611)
>        at org.jboss.netty.channel.Channels.write(Channels.java:578)
>        at
> org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259)
>        at