Also, if you end up choosing to use Kafka and persisting your messages into
Hadoop, then you should take a look at
is also from LinkedIn).
If you do things the LinkedIn way right from the start (i.e.: using the
AVRO-1124 schema repo and encoding time in a standard way in a header
contained in all your schemas), then you can use Camus pretty much out of
the box without any tweaking, and the solution you'll get is very flexible
/ extensible (regarding the ability to evolve your schemas gracefully,
letting Camus discover new topics to persist automatically, etc).
For us it was a little more complicated since we had some legacy stuff that
was not exactly how Camus expected it, but it wasn't that complicated to
On Thu, Jun 6, 2013 at 2:51 PM, Felix GV <[EMAIL PROTECTED]> wrote:
> You can serialize avro messages into json or binary format, and then pass
> them around to anything else (send them over HTTP, publish them to a
> message broker system like Kafka or Flume, write them directly into a data
> store, etc.). You can forget about the avro RPC, as it's just one way
> amongst many of doing this.
> You do need to manage schemas properly though. The easy way is to hardcode
> your schema on both ends, but then that makes it harder to evolve schemas
> (which avro can do very well otherwise). If you send single serialized avro
> messages around through a message broker system, then you should definitely
> consider using a version number for your schema at the beginning of the
> message, as Martin suggested. Then you can look up what schema each version
> number represents with something like the versioned schema repo in
> AVRO-1124 <https://issues.apache.org/jira/browse/AVRO-1124>.
> On Tue, Jun 4, 2013 at 11:10 PM, Mark <[EMAIL PROTECTED]> wrote:
>> I have a question. Say I want to use AVRO as my serialization format to
>> speak between service applications. Do I need to use AVRO RPC for this or
>> can I just exchange AVRO messages over HTTP?
>> Also whats the difference between an IPC client and an HTTP IPC client?
>> On May 29, 2013, at 8:02 PM, Mike Percy <[EMAIL PROTECTED]> wrote:
>> There is no Ruby support for the Netty Avro RPC protocol that I know of.
>> But I'm not sure why that matters, other than the fact that the Flume
>> Thrift support it's not in an official release yet.
>> You could also take a look at the Flume HTTP source for a REST-based
>> interface, but to accept binary data instead of JSON (the default) you
>> would need to write a small bit of Java code and plug that in.
>> Make sure you differentiate between using Avro as a data storage format
>> and as an RPC mechanism. They are two very different things and don't need
>> to be tied together. Today, the data storage aspect is more mature and has
>> much wider language support.
>> On Wed, May 29, 2013 at 9:30 AM, Mark <[EMAIL PROTECTED]> wrote:
>>> So basically Avro RPC is out of the question? Instead I would need to
>>> Avro Message -> Thrift -> Flume? Is that along the right lines or am I
>>> missing something?
>>> On May 28, 2013, at 5:02 PM, Mike Percy <[EMAIL PROTECTED]> wrote:
>>> Regarding Ruby support, we recently added support for Thrift RPC, so you
>>> can now send messages to Flume via Ruby and other non-JVM languages. We
>>> don't have out-of-the-box client APIs for those yet but would be happy to
>>> accept patches for it :)