Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # user - Python/Java/Ruby RPC interop issues

Stephen Abrams 2013-06-20, 17:01
Stefan Krawczyk 2013-06-25, 00:30
Copy link to this message
Re: Python/Java/Ruby RPC interop issues
Stephen Abrams 2013-07-13, 17:45
Hi Stefan,

Thanks for the response (and apologies for the delay). The Thrift
workaround (as you say, introduced in 1.4) is an interesting workaround. We
are  primarily a Java shop, but some of our tests are written in Python,
which I was using for some quick exploration. If I get ambitious enough I
may try Jython<->Python communication (since I don't want to use Flume for
everything), and post back the results.

On Mon, Jun 24, 2013 at 8:30 PM, Stefan Krawczyk <[EMAIL PROTECTED]>wrote:

> Hi Steve,
> I'm new to the Avro community as well, but it just seems that nobody
> really cares about Avro support outside of Java/C++...
> The issue you're seeing is because the python & ruby implementations are
> using the HTTPTransceiver, whereas the Java server is using the
> NettyTransceiver. They dont' talk the same language. I'm as surprised as
> you are that such a change would be introduced without the other languages
> having support (or providing documentation mentioning that they don't work).
> I asked this list earlier what it would take to get python to talk netty
> but nobody responded...
> We're also using Flume, and the only reasonable option is to use Thrift
> RPC & the Thrift Source (look at 1.4 branch of code) as the ingestion point
> from our app. From there the rest of the flume pipeline can use Avro RPC.
> The other option is to use the legacy avro source, but that source is
> effectively deprecated...
> Hope that helps.
> Cheers,
> Stefan
> On Thu, Jun 20, 2013 at 10:01 AM, Stephen Abrams <[EMAIL PROTECTED]
> > wrote:
>> Hello!
>> New to the Avro world. I have been evaluating Flume and testing various
>> capabilities. I have run into an issue with the Avro RPC piece. So far, I
>> have found that the Flume Avro RPC java client works with a Flume Avro
>> Source as expected (I believe these implementations use Netty). However,
>> when I tried using a Ruby RPC client to communicate with the Flume Avro
>> Source, there was a failure and the source reported:
>> org.apache.avro.AvroRuntimeException: Excessively large list allocation
>> request detected: 539959368 items! Connection closed.
>> I then tried this with a Python RPC client and had a similar issue. I
>> suspected client/server Avro interop issues, so then verified that a Python
>> Avro RPC client and Python Avro RPC server communicate correctly (using
>> https://github.com/phunt/avro-rpc-quickstart). Also verified that Ruby
>> Avro RPC client and Ruby Avro RPC server communicate correctly. However,
>> using the Python Avro RPC client with Ruby Avro RPC server fails (client
>> hangs), and Ruby client with Python server also fails (again, client hangs).
>> I know I'm not providing exhaustive info here, but I'm surprised to find
>> out-of-the-box interop problems. Do folks regularly use these
>> client/servers in various mixes (Java/Ruby/Python)? Are there particular
>> gotchas or configurations that are not obvious?
>> Thank you,
>> Steve