Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Python/Java/Ruby RPC interop issues


+
Stephen Abrams 2013-06-20, 17:01
Copy link to this message
-
Re: Python/Java/Ruby RPC interop issues
Hi Steve,

I'm new to the Avro community as well, but it just seems that nobody really
cares about Avro support outside of Java/C++...

The issue you're seeing is because the python & ruby implementations are
using the HTTPTransceiver, whereas the Java server is using the
NettyTransceiver. They dont' talk the same language. I'm as surprised as
you are that such a change would be introduced without the other languages
having support (or providing documentation mentioning that they don't work).

I asked this list earlier what it would take to get python to talk netty
but nobody responded...

We're also using Flume, and the only reasonable option is to use Thrift RPC
& the Thrift Source (look at 1.4 branch of code) as the ingestion point
from our app. From there the rest of the flume pipeline can use Avro RPC.
The other option is to use the legacy avro source, but that source is
effectively deprecated...

Hope that helps.

Cheers,

Stefan

On Thu, Jun 20, 2013 at 10:01 AM, Stephen Abrams
<[EMAIL PROTECTED]>wrote:

> Hello!
>
> New to the Avro world. I have been evaluating Flume and testing various
> capabilities. I have run into an issue with the Avro RPC piece. So far, I
> have found that the Flume Avro RPC java client works with a Flume Avro
> Source as expected (I believe these implementations use Netty). However,
> when I tried using a Ruby RPC client to communicate with the Flume Avro
> Source, there was a failure and the source reported:
>
> org.apache.avro.AvroRuntimeException: Excessively large list allocation
> request detected: 539959368 items! Connection closed.
>
> I then tried this with a Python RPC client and had a similar issue. I
> suspected client/server Avro interop issues, so then verified that a Python
> Avro RPC client and Python Avro RPC server communicate correctly (using
> https://github.com/phunt/avro-rpc-quickstart). Also verified that Ruby
> Avro RPC client and Ruby Avro RPC server communicate correctly. However,
> using the Python Avro RPC client with Ruby Avro RPC server fails (client
> hangs), and Ruby client with Python server also fails (again, client hangs).
>
> I know I'm not providing exhaustive info here, but I'm surprised to find
> out-of-the-box interop problems. Do folks regularly use these
> client/servers in various mixes (Java/Ruby/Python)? Are there particular
> gotchas or configurations that are not obvious?
>
> Thank you,
> Steve
>
+
Stephen Abrams 2013-07-13, 17:45
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB