Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # user - avro-rpc: running multiple threads on the netty server


+
Matt Corgan 2013-01-28, 00:16
+
Matt Corgan 2013-01-31, 02:38
+
Philip Zeyliger 2013-01-31, 03:04
+
Philip Zeyliger 2013-01-31, 03:04
Copy link to this message
-
Re: avro-rpc: running multiple threads on the netty server
Matt Corgan 2013-01-31, 03:27
Thanks Philip, I plan to try the http approach as well for clients that
can't use Netty

..sent from my phone
On Jan 30, 2013 7:05 PM, "Philip Zeyliger" <[EMAIL PROTECTED]> wrote:

> Matt,
>
> I misread your question.  The multiple-thread approach is with the
> Jetty/HTTP mechanism, not the Netty one.  Sorry 'bout the noise.
>
>
> On Wed, Jan 30, 2013 at 7:04 PM, Philip Zeyliger <[EMAIL PROTECTED]>wrote:
>
>> Hi Matt,
>>
>> I do this with some frequency.  The trick is to use Jetty manually to
>> create an appropriate jetty.Server, and then start it.  The snippets below
>> should help.
>>
>>  -- Philip
>>
>> import javax.servlet.Servlet;
>>
>> import org.apache.avro.ipc.ResponderServlet;
>> import org.apache.avro.ipc.specific.SpecificResponder;
>> import org.mortbay.jetty.Server;
>> import org.mortbay.jetty.servlet.Context;
>> import org.mortbay.jetty.servlet.ServletHolder;
>> import org.mortbay.thread.QueuedThreadPool;
>>
>>   public static Server createAvroServer(int port, String name, int
>> maxThreads,
>>                                         int maxIdleTimeMs,
>>                                         SpecificResponder responder)
>>                                         throws EnterpriseServiceException
>> {
>>     try {
>>       Server httpServer = new Server(port);
>>       QueuedThreadPool qtp = new QueuedThreadPool();
>>       // QueuedThreadPool is jetty's thread pool implementation;
>>       // this lets us give it a name.
>>       qtp.setName(name);
>>       qtp.setDaemon(true);
>>       qtp.setMaxThreads(maxThreads);
>>       qtp.setMaxIdleTimeMs(maxIdleTimeMs);
>>       httpServer.setThreadPool(qtp);
>>       Servlet servlet = new ResponderServlet(responder);
>>       new Context(httpServer, "/").addServlet(new ServletHolder(servlet),
>> "/*");
>>       return httpServer;
>>     } catch (Exception e) {
>>       throw new EnterpriseServiceException(e);
>>     }
>>   }
>>
>>    // main:
>>       SpecificResponder responder = new SpecificResponder(
>>               XXX.class,
>>               new XXXImpl());
>>       server = createAvroServer(listenPort, "somename",
>>                    maxThreads, maxIdleTimeMs, responder);
>>        server.start();
>>
>>
>> On Wed, Jan 30, 2013 at 6:38 PM, Matt Corgan <[EMAIL PROTECTED]> wrote:
>>
>>> hmm - i haven't heard any suggestions in the past few days... maybe a
>>> really short version of the question:  Can an Avro-rpc Netty server process
>>> requests in parallel?  Will it still process in parallel if they're from
>>> the same client?
>>>
>>> Thanks much,
>>> Matt
>>>
>>>
>>> On Sun, Jan 27, 2013 at 4:16 PM, Matt Corgan <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hi,
>>>> I'm trying to familiarize myself with avro-rpc by running and tweaking
>>>> the example at https://github.com/jbaldassari/Avro-RPC.  By the way,
>>>> thanks for creating this example James.
>>>>
>>>> I'm using the BidderTest.sendBidRequestWithCallback method and am
>>>> trying to get the server to process the requests in parallel, which is
>>>> something i'll need in my application.  I use the DelayInjectingBidder with
>>>> delay of 90ms so i can watch what's happening on the server.  By debugging
>>>> the client, I see that all requests do in fact make it to the server
>>>> without blocking, but watching the server log i see that each request is
>>>> processed sequentially.  So it takes ~9s to process 100 messages.
>>>>
>>>> Digging into the NettyServer constructor, I see the default
>>>> ChannelFactory uses Executors.newCachedThreadPool() for the Boss and Worker
>>>> executors.  I tried overriding the worker with
>>>> Executors.newFixedThreadPool(8) but didn't notice any difference.  Looking
>>>> at the threads running in Eclipse Debug mode, i see the following threads
>>>> which confirms that there is only one worker:
>>>>
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner at
>>>> localhost:50255
>>>> Thread [main] (Running)
>>>>  Thread [ReaderThread] (Running)
>>>> Thread [New I/O server boss #2 ([id: 0x6a5dd151,