Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - avro-rpc: running multiple threads on the netty server


Copy link to this message
-
Re: avro-rpc: running multiple threads on the netty server
Matt Corgan 2013-01-31, 02:38
hmm - i haven't heard any suggestions in the past few days... maybe a
really short version of the question:  Can an Avro-rpc Netty server process
requests in parallel?  Will it still process in parallel if they're from
the same client?

Thanks much,
Matt
On Sun, Jan 27, 2013 at 4:16 PM, Matt Corgan <[EMAIL PROTECTED]> wrote:

> Hi,
> I'm trying to familiarize myself with avro-rpc by running and tweaking the
> example at https://github.com/jbaldassari/Avro-RPC.  By the way, thanks
> for creating this example James.
>
> I'm using the BidderTest.sendBidRequestWithCallback method and am trying
> to get the server to process the requests in parallel, which is something
> i'll need in my application.  I use the DelayInjectingBidder with delay of
> 90ms so i can watch what's happening on the server.  By debugging the
> client, I see that all requests do in fact make it to the server without
> blocking, but watching the server log i see that each request is processed
> sequentially.  So it takes ~9s to process 100 messages.
>
> Digging into the NettyServer constructor, I see the default ChannelFactory
> uses Executors.newCachedThreadPool() for the Boss and Worker executors.  I
> tried overriding the worker with Executors.newFixedThreadPool(8) but didn't
> notice any difference.  Looking at the threads running in Eclipse Debug
> mode, i see the following threads which confirms that there is only one
> worker:
>
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner at localhost:50255
> Thread [main] (Running)
>  Thread [ReaderThread] (Running)
> Thread [New I/O server boss #2 ([id: 0x6a5dd151, /0:0:0:0:0:0:0:0:45633])]
> (Running)
>  Thread [Avro NettyTransceiver Boss 1] (Running)
> Thread [New I/O server worker #2-1] (Running)
>  Thread [New I/O client worker #1-1] (Running)
>
> I also see that the NettyTransceiver has configurable thread pools, but
> I'm not clear on what the Transceiver's role is (client side thing?), and
> fiddling with its ChannelFactory is causing exceptions.
>
> Is is possible that the behavior of the BidderTest client doesn't trigger
> the multi-threaded server even though it's configured correctly, or that
> the server processes all requests from a single client sequentially?
>
> Sidenote: i also tried using avro version 1.7.3 with the same results,
> although the above thread pools had different names.
>
> I wonder if someone could shed some light on what the difference is
> between the Server and Transceiver thread pools and how to configure them
> for multi-threaded server processing.
>
> Thanks,
> Matt
>
>