Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # user - Avro RPC: Python to Java isn't working for me...


+
Stefan Krawczyk 2013-05-22, 03:18
+
Atin Sood 2013-05-22, 11:11
+
Stefan Krawczyk 2013-05-22, 15:55
+
Atin Sood 2013-05-22, 23:22
Copy link to this message
-
Re: Avro RPC: Python to Java isn't working for me...
Stefan Krawczyk 2013-05-22, 23:36
Yes agreed, that would be a good approach to take, however before I go try
to figure out how to write a NettyTransceiver in python, I'd like to get
some community input on so that:
 - I understand why there wasn't one built in the first place, and thus
answer is it actually possible to build that in python
 - I'm not duplicating effort if there is somebody already working on it

So any NettyServer & NettyTransceiver experts in the house?

:)

Cheers,

Stefan

On Wed, May 22, 2013 at 4:22 PM, Atin Sood <[EMAIL PROTECTED]> wrote:

> Sure Stefan
>
> I am a newbie to both python and avro and have limited experience in
> networking.
>
> But if I am not wrong the problem is at python client end as I am guessing
> you must be using s.th like
>
>  client code - attach to the server and send a message
>
>     client = ipc.HTTPTransceiver(server_addr[0], server_addr[1])
>     requestor = ipc.Requestor(PROTOCOL, client)
>
>
> So even though you can go ahead and switch to NettyServer in java you will be limited to use httpserver because your python client uses http client
>
> https://github.com/phunt/avro-rpc-quickstart/blob/master/src/main/java/example/Main.java
>
> I guess the best way to get around this will be to look at source code of avro python client ipc.py file
>
> and add a new implementation besides the one that comes out of the box.
>
> class HTTPTransceiver(object):
>   """
>   A simple HTTP-based transceiver implementation.
>   Useful for clients but not for servers
>   """
>
>
> That said, again I am new to the whole thing so I might be totally wrong :)
>
>
> --
> Atin Sood
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On Wednesday, May 22, 2013 at 11:55 AM, Stefan Krawczyk wrote:
>
> Hi Atin,
>
> Thanks for the response. Yes I understand I could use HTTPServer on the
> java side and things would work. However I'm after a solution where I can
> still have the java side use the NettyServer.
>
> Cheers,
>
> Stefan
>
>
> On Wed, May 22, 2013 at 4:11 AM, Atin Sood <[EMAIL PROTECTED]> wrote:
>
> You can try looking into something that I wrote as an example
>
>
> https://github.com/atinsood/HESDataAnalyticsFinalProject/tree/master/javaXPython
>
> https://github.com/atinsood/HESDataAnalyticsFinalProject#javaxpython
>
> --
> Atin Sood
> Sent with Sparrow <http://www.sparrowmailapp.com/?sig>
>
> On Tuesday, May 21, 2013 at 11:18 PM, Stefan Krawczyk wrote:
>
> Hi,
>
> I am trying to use Avro RPC and have a python client talk to a java
> server, using the avro-rpc-quickstart<https://github.com/phunt/avro-rpc-quickstart> on
> github as a base (I made sure the avro version being pulled in was 1.7.4).
> However when I get my python client to talk to the java server I see this
> error:
>
> 2013-05-20 19:38:32,512 (pool-5-thread-2) [WARN -
> org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught(NettyServer.java:201)]
> Unexpected exception from downstream.
> org.apache.avro.AvroRuntimeException: Excessively large list allocation
> request detected: 539959368 items! Connection closed.
> at
> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:167)
>  at
> org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:139)
> at
> org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:286)
>  at
> org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
> at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
>  at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
>  at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
>  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
+
Atin Sood 2013-05-29, 00:07
+
Stefan Krawczyk 2013-05-29, 04:12