Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> avro-rpc: running multiple threads on the netty server


Copy link to this message
-
Re: avro-rpc: running multiple threads on the netty server
hmm - i haven't heard any suggestions in the past few days... maybe a
really short version of the question:  Can an Avro-rpc Netty server process
requests in parallel?  Will it still process in parallel if they're from
the same client?

Thanks much,
Matt
On Sun, Jan 27, 2013 at 4:16 PM, Matt Corgan <[EMAIL PROTECTED]> wrote:

> Hi,
> I'm trying to familiarize myself with avro-rpc by running and tweaking the
> example at https://github.com/jbaldassari/Avro-RPC.  By the way, thanks
> for creating this example James.
>
> I'm using the BidderTest.sendBidRequestWithCallback method and am trying
> to get the server to process the requests in parallel, which is something
> i'll need in my application.  I use the DelayInjectingBidder with delay of
> 90ms so i can watch what's happening on the server.  By debugging the
> client, I see that all requests do in fact make it to the server without
> blocking, but watching the server log i see that each request is processed
> sequentially.  So it takes ~9s to process 100 messages.
>
> Digging into the NettyServer constructor, I see the default ChannelFactory
> uses Executors.newCachedThreadPool() for the Boss and Worker executors.  I
> tried overriding the worker with Executors.newFixedThreadPool(8) but didn't
> notice any difference.  Looking at the threads running in Eclipse Debug
> mode, i see the following threads which confirms that there is only one
> worker:
>
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner at localhost:50255
> Thread [main] (Running)
>  Thread [ReaderThread] (Running)
> Thread [New I/O server boss #2 ([id: 0x6a5dd151, /0:0:0:0:0:0:0:0:45633])]
> (Running)
>  Thread [Avro NettyTransceiver Boss 1] (Running)
> Thread [New I/O server worker #2-1] (Running)
>  Thread [New I/O client worker #1-1] (Running)
>
> I also see that the NettyTransceiver has configurable thread pools, but
> I'm not clear on what the Transceiver's role is (client side thing?), and
> fiddling with its ChannelFactory is causing exceptions.
>
> Is is possible that the behavior of the BidderTest client doesn't trigger
> the multi-threaded server even though it's configured correctly, or that
> the server processes all requests from a single client sequentially?
>
> Sidenote: i also tried using avro version 1.7.3 with the same results,
> although the above thread pools had different names.
>
> I wonder if someone could shed some light on what the difference is
> between the Server and Transceiver thread pools and how to configure them
> for multi-threaded server processing.
>
> Thanks,
> Matt
>
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB