Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> avro-rpc: running multiple threads on the netty server


Copy link to this message
-
Re: avro-rpc: running multiple threads on the netty server
Matt,

I misread your question.  The multiple-thread approach is with the
Jetty/HTTP mechanism, not the Netty one.  Sorry 'bout the noise.
On Wed, Jan 30, 2013 at 7:04 PM, Philip Zeyliger <[EMAIL PROTECTED]>wrote:

> Hi Matt,
>
> I do this with some frequency.  The trick is to use Jetty manually to
> create an appropriate jetty.Server, and then start it.  The snippets below
> should help.
>
>  -- Philip
>
> import javax.servlet.Servlet;
>
> import org.apache.avro.ipc.ResponderServlet;
> import org.apache.avro.ipc.specific.SpecificResponder;
> import org.mortbay.jetty.Server;
> import org.mortbay.jetty.servlet.Context;
> import org.mortbay.jetty.servlet.ServletHolder;
> import org.mortbay.thread.QueuedThreadPool;
>
>   public static Server createAvroServer(int port, String name, int
> maxThreads,
>                                         int maxIdleTimeMs,
>                                         SpecificResponder responder)
>                                         throws EnterpriseServiceException {
>     try {
>       Server httpServer = new Server(port);
>       QueuedThreadPool qtp = new QueuedThreadPool();
>       // QueuedThreadPool is jetty's thread pool implementation;
>       // this lets us give it a name.
>       qtp.setName(name);
>       qtp.setDaemon(true);
>       qtp.setMaxThreads(maxThreads);
>       qtp.setMaxIdleTimeMs(maxIdleTimeMs);
>       httpServer.setThreadPool(qtp);
>       Servlet servlet = new ResponderServlet(responder);
>       new Context(httpServer, "/").addServlet(new ServletHolder(servlet),
> "/*");
>       return httpServer;
>     } catch (Exception e) {
>       throw new EnterpriseServiceException(e);
>     }
>   }
>
>    // main:
>       SpecificResponder responder = new SpecificResponder(
>               XXX.class,
>               new XXXImpl());
>       server = createAvroServer(listenPort, "somename",
>                    maxThreads, maxIdleTimeMs, responder);
>        server.start();
>
>
> On Wed, Jan 30, 2013 at 6:38 PM, Matt Corgan <[EMAIL PROTECTED]> wrote:
>
>> hmm - i haven't heard any suggestions in the past few days... maybe a
>> really short version of the question:  Can an Avro-rpc Netty server process
>> requests in parallel?  Will it still process in parallel if they're from
>> the same client?
>>
>> Thanks much,
>> Matt
>>
>>
>> On Sun, Jan 27, 2013 at 4:16 PM, Matt Corgan <[EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>> I'm trying to familiarize myself with avro-rpc by running and tweaking
>>> the example at https://github.com/jbaldassari/Avro-RPC.  By the way,
>>> thanks for creating this example James.
>>>
>>> I'm using the BidderTest.sendBidRequestWithCallback method and am trying
>>> to get the server to process the requests in parallel, which is something
>>> i'll need in my application.  I use the DelayInjectingBidder with delay of
>>> 90ms so i can watch what's happening on the server.  By debugging the
>>> client, I see that all requests do in fact make it to the server without
>>> blocking, but watching the server log i see that each request is processed
>>> sequentially.  So it takes ~9s to process 100 messages.
>>>
>>> Digging into the NettyServer constructor, I see the default
>>> ChannelFactory uses Executors.newCachedThreadPool() for the Boss and Worker
>>> executors.  I tried overriding the worker with
>>> Executors.newFixedThreadPool(8) but didn't notice any difference.  Looking
>>> at the threads running in Eclipse Debug mode, i see the following threads
>>> which confirms that there is only one worker:
>>>
>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner at localhost:50255
>>> Thread [main] (Running)
>>>  Thread [ReaderThread] (Running)
>>> Thread [New I/O server boss #2 ([id: 0x6a5dd151,
>>> /0:0:0:0:0:0:0:0:45633])] (Running)
>>>  Thread [Avro NettyTransceiver Boss 1] (Running)
>>> Thread [New I/O server worker #2-1] (Running)
>>>  Thread [New I/O client worker #1-1] (Running)
>>>
>>> I also see that the NettyTransceiver has configurable thread pools, but