Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> avro-rpc: running multiple threads on the netty server


Copy link to this message
-
avro-rpc: running multiple threads on the netty server
Hi,
I'm trying to familiarize myself with avro-rpc by running and tweaking the
example at https://github.com/jbaldassari/Avro-RPC.  By the way, thanks for
creating this example James.

I'm using the BidderTest.sendBidRequestWithCallback method and am trying to
get the server to process the requests in parallel, which is something i'll
need in my application.  I use the DelayInjectingBidder with delay of 90ms
so i can watch what's happening on the server.  By debugging the client, I
see that all requests do in fact make it to the server without blocking,
but watching the server log i see that each request is processed
sequentially.  So it takes ~9s to process 100 messages.

Digging into the NettyServer constructor, I see the default ChannelFactory
uses Executors.newCachedThreadPool() for the Boss and Worker executors.  I
tried overriding the worker with Executors.newFixedThreadPool(8) but didn't
notice any difference.  Looking at the threads running in Eclipse Debug
mode, i see the following threads which confirms that there is only one
worker:

org.eclipse.jdt.internal.junit.runner.RemoteTestRunner at localhost:50255
Thread [main] (Running)
Thread [ReaderThread] (Running)
Thread [New I/O server boss #2 ([id: 0x6a5dd151, /0:0:0:0:0:0:0:0:45633])]
(Running)
Thread [Avro NettyTransceiver Boss 1] (Running)
Thread [New I/O server worker #2-1] (Running)
Thread [New I/O client worker #1-1] (Running)

I also see that the NettyTransceiver has configurable thread pools, but I'm
not clear on what the Transceiver's role is (client side thing?), and
fiddling with its ChannelFactory is causing exceptions.

Is is possible that the behavior of the BidderTest client doesn't trigger
the multi-threaded server even though it's configured correctly, or that
the server processes all requests from a single client sequentially?

Sidenote: i also tried using avro version 1.7.3 with the same results,
although the above thread pools had different names.

I wonder if someone could shed some light on what the difference is between
the Server and Transceiver thread pools and how to configure them for
multi-threaded server processing.

Thanks,
Matt
+
Matt Corgan 2013-01-31, 02:38
+
Philip Zeyliger 2013-01-31, 03:04
+
Philip Zeyliger 2013-01-31, 03:04
+
Matt Corgan 2013-01-31, 03:27