Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Python/Java/Ruby RPC interop issues

Stephen Abrams 2013-06-20, 17:01
Stefan Krawczyk 2013-06-25, 00:30
Copy link to this message
Re: Python/Java/Ruby RPC interop issues
Hi Stefan,

Thanks for the response (and apologies for the delay). The Thrift
workaround (as you say, introduced in 1.4) is an interesting workaround. We
are  primarily a Java shop, but some of our tests are written in Python,
which I was using for some quick exploration. If I get ambitious enough I
may try Jython<->Python communication (since I don't want to use Flume for
everything), and post back the results.

On Mon, Jun 24, 2013 at 8:30 PM, Stefan Krawczyk <[EMAIL PROTECTED]>wrote:

> Hi Steve,
> I'm new to the Avro community as well, but it just seems that nobody
> really cares about Avro support outside of Java/C++...
> The issue you're seeing is because the python & ruby implementations are
> using the HTTPTransceiver, whereas the Java server is using the
> NettyTransceiver. They dont' talk the same language. I'm as surprised as
> you are that such a change would be introduced without the other languages
> having support (or providing documentation mentioning that they don't work).
> I asked this list earlier what it would take to get python to talk netty
> but nobody responded...
> We're also using Flume, and the only reasonable option is to use Thrift
> RPC & the Thrift Source (look at 1.4 branch of code) as the ingestion point
> from our app. From there the rest of the flume pipeline can use Avro RPC.
> The other option is to use the legacy avro source, but that source is
> effectively deprecated...
> Hope that helps.
> Cheers,
> Stefan
> On Thu, Jun 20, 2013 at 10:01 AM, Stephen Abrams <[EMAIL PROTECTED]
> > wrote:
>> Hello!
>> New to the Avro world. I have been evaluating Flume and testing various
>> capabilities. I have run into an issue with the Avro RPC piece. So far, I
>> have found that the Flume Avro RPC java client works with a Flume Avro
>> Source as expected (I believe these implementations use Netty). However,
>> when I tried using a Ruby RPC client to communicate with the Flume Avro
>> Source, there was a failure and the source reported:
>> org.apache.avro.AvroRuntimeException: Excessively large list allocation
>> request detected: 539959368 items! Connection closed.
>> I then tried this with a Python RPC client and had a similar issue. I
>> suspected client/server Avro interop issues, so then verified that a Python
>> Avro RPC client and Python Avro RPC server communicate correctly (using
>> https://github.com/phunt/avro-rpc-quickstart). Also verified that Ruby
>> Avro RPC client and Ruby Avro RPC server communicate correctly. However,
>> using the Python Avro RPC client with Ruby Avro RPC server fails (client
>> hangs), and Ruby client with Python server also fails (again, client hangs).
>> I know I'm not providing exhaustive info here, but I'm surprised to find
>> out-of-the-box interop problems. Do folks regularly use these
>> client/servers in various mixes (Java/Ruby/Python)? Are there particular
>> gotchas or configurations that are not obvious?
>> Thank you,
>> Steve