Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> avro-rpc: running multiple threads on the netty server


+
Matt Corgan 2013-01-28, 00:16
+
Matt Corgan 2013-01-31, 02:38
Copy link to this message
-
Re: avro-rpc: running multiple threads on the netty server
Hi Matt,

I do this with some frequency.  The trick is to use Jetty manually to
create an appropriate jetty.Server, and then start it.  The snippets below
should help.

-- Philip

import javax.servlet.Servlet;

import org.apache.avro.ipc.ResponderServlet;
import org.apache.avro.ipc.specific.SpecificResponder;
import org.mortbay.jetty.Server;
import org.mortbay.jetty.servlet.Context;
import org.mortbay.jetty.servlet.ServletHolder;
import org.mortbay.thread.QueuedThreadPool;

  public static Server createAvroServer(int port, String name, int
maxThreads,
                                        int maxIdleTimeMs,
                                        SpecificResponder responder)
                                        throws EnterpriseServiceException {
    try {
      Server httpServer = new Server(port);
      QueuedThreadPool qtp = new QueuedThreadPool();
      // QueuedThreadPool is jetty's thread pool implementation;
      // this lets us give it a name.
      qtp.setName(name);
      qtp.setDaemon(true);
      qtp.setMaxThreads(maxThreads);
      qtp.setMaxIdleTimeMs(maxIdleTimeMs);
      httpServer.setThreadPool(qtp);
      Servlet servlet = new ResponderServlet(responder);
      new Context(httpServer, "/").addServlet(new ServletHolder(servlet),
"/*");
      return httpServer;
    } catch (Exception e) {
      throw new EnterpriseServiceException(e);
    }
  }

   // main:
      SpecificResponder responder = new SpecificResponder(
              XXX.class,
              new XXXImpl());
      server = createAvroServer(listenPort, "somename",
                   maxThreads, maxIdleTimeMs, responder);
      server.start();
On Wed, Jan 30, 2013 at 6:38 PM, Matt Corgan <[EMAIL PROTECTED]> wrote:

> hmm - i haven't heard any suggestions in the past few days... maybe a
> really short version of the question:  Can an Avro-rpc Netty server process
> requests in parallel?  Will it still process in parallel if they're from
> the same client?
>
> Thanks much,
> Matt
>
>
> On Sun, Jan 27, 2013 at 4:16 PM, Matt Corgan <[EMAIL PROTECTED]> wrote:
>
>> Hi,
>> I'm trying to familiarize myself with avro-rpc by running and tweaking
>> the example at https://github.com/jbaldassari/Avro-RPC.  By the way,
>> thanks for creating this example James.
>>
>> I'm using the BidderTest.sendBidRequestWithCallback method and am trying
>> to get the server to process the requests in parallel, which is something
>> i'll need in my application.  I use the DelayInjectingBidder with delay of
>> 90ms so i can watch what's happening on the server.  By debugging the
>> client, I see that all requests do in fact make it to the server without
>> blocking, but watching the server log i see that each request is processed
>> sequentially.  So it takes ~9s to process 100 messages.
>>
>> Digging into the NettyServer constructor, I see the default
>> ChannelFactory uses Executors.newCachedThreadPool() for the Boss and Worker
>> executors.  I tried overriding the worker with
>> Executors.newFixedThreadPool(8) but didn't notice any difference.  Looking
>> at the threads running in Eclipse Debug mode, i see the following threads
>> which confirms that there is only one worker:
>>
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner at localhost:50255
>> Thread [main] (Running)
>>  Thread [ReaderThread] (Running)
>> Thread [New I/O server boss #2 ([id: 0x6a5dd151,
>> /0:0:0:0:0:0:0:0:45633])] (Running)
>>  Thread [Avro NettyTransceiver Boss 1] (Running)
>> Thread [New I/O server worker #2-1] (Running)
>>  Thread [New I/O client worker #1-1] (Running)
>>
>> I also see that the NettyTransceiver has configurable thread pools, but
>> I'm not clear on what the Transceiver's role is (client side thing?), and
>> fiddling with its ChannelFactory is causing exceptions.
>>
>> Is is possible that the behavior of the BidderTest client doesn't trigger
>> the multi-threaded server even though it's configured correctly, or that
>> the server processes all requests from a single client sequentially?
+
Philip Zeyliger 2013-01-31, 03:04
+
Matt Corgan 2013-01-31, 03:27
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB