Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> avro-rpc: running multiple threads on the netty server


Copy link to this message
-
Re: avro-rpc: running multiple threads on the netty server
Thanks Philip, I plan to try the http approach as well for clients that
can't use Netty

..sent from my phone
On Jan 30, 2013 7:05 PM, "Philip Zeyliger" <[EMAIL PROTECTED]> wrote:

> Matt,
>
> I misread your question.  The multiple-thread approach is with the
> Jetty/HTTP mechanism, not the Netty one.  Sorry 'bout the noise.
>
>
> On Wed, Jan 30, 2013 at 7:04 PM, Philip Zeyliger <[EMAIL PROTECTED]>wrote:
>
>> Hi Matt,
>>
>> I do this with some frequency.  The trick is to use Jetty manually to
>> create an appropriate jetty.Server, and then start it.  The snippets below
>> should help.
>>
>>  -- Philip
>>
>> import javax.servlet.Servlet;
>>
>> import org.apache.avro.ipc.ResponderServlet;
>> import org.apache.avro.ipc.specific.SpecificResponder;
>> import org.mortbay.jetty.Server;
>> import org.mortbay.jetty.servlet.Context;
>> import org.mortbay.jetty.servlet.ServletHolder;
>> import org.mortbay.thread.QueuedThreadPool;
>>
>>   public static Server createAvroServer(int port, String name, int
>> maxThreads,
>>                                         int maxIdleTimeMs,
>>                                         SpecificResponder responder)
>>                                         throws EnterpriseServiceException
>> {
>>     try {
>>       Server httpServer = new Server(port);
>>       QueuedThreadPool qtp = new QueuedThreadPool();
>>       // QueuedThreadPool is jetty's thread pool implementation;
>>       // this lets us give it a name.
>>       qtp.setName(name);
>>       qtp.setDaemon(true);
>>       qtp.setMaxThreads(maxThreads);
>>       qtp.setMaxIdleTimeMs(maxIdleTimeMs);
>>       httpServer.setThreadPool(qtp);
>>       Servlet servlet = new ResponderServlet(responder);
>>       new Context(httpServer, "/").addServlet(new ServletHolder(servlet),
>> "/*");
>>       return httpServer;
>>     } catch (Exception e) {
>>       throw new EnterpriseServiceException(e);
>>     }
>>   }
>>
>>    // main:
>>       SpecificResponder responder = new SpecificResponder(
>>               XXX.class,
>>               new XXXImpl());
>>       server = createAvroServer(listenPort, "somename",
>>                    maxThreads, maxIdleTimeMs, responder);
>>        server.start();
>>
>>
>> On Wed, Jan 30, 2013 at 6:38 PM, Matt Corgan <[EMAIL PROTECTED]> wrote:
>>
>>> hmm - i haven't heard any suggestions in the past few days... maybe a
>>> really short version of the question:  Can an Avro-rpc Netty server process
>>> requests in parallel?  Will it still process in parallel if they're from
>>> the same client?
>>>
>>> Thanks much,
>>> Matt
>>>
>>>
>>> On Sun, Jan 27, 2013 at 4:16 PM, Matt Corgan <[EMAIL PROTECTED]>wrote:
>>>
>>>> Hi,
>>>> I'm trying to familiarize myself with avro-rpc by running and tweaking
>>>> the example at https://github.com/jbaldassari/Avro-RPC.  By the way,
>>>> thanks for creating this example James.
>>>>
>>>> I'm using the BidderTest.sendBidRequestWithCallback method and am
>>>> trying to get the server to process the requests in parallel, which is
>>>> something i'll need in my application.  I use the DelayInjectingBidder with
>>>> delay of 90ms so i can watch what's happening on the server.  By debugging
>>>> the client, I see that all requests do in fact make it to the server
>>>> without blocking, but watching the server log i see that each request is
>>>> processed sequentially.  So it takes ~9s to process 100 messages.
>>>>
>>>> Digging into the NettyServer constructor, I see the default
>>>> ChannelFactory uses Executors.newCachedThreadPool() for the Boss and Worker
>>>> executors.  I tried overriding the worker with
>>>> Executors.newFixedThreadPool(8) but didn't notice any difference.  Looking
>>>> at the threads running in Eclipse Debug mode, i see the following threads
>>>> which confirms that there is only one worker:
>>>>
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner at
>>>> localhost:50255
>>>> Thread [main] (Running)
>>>>  Thread [ReaderThread] (Running)
>>>> Thread [New I/O server boss #2 ([id: 0x6a5dd151,
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB