Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Avro serialization


+
Mark 2013-08-20, 14:08
+
Guozhang Wang 2013-08-20, 16:19
+
Mark 2013-08-20, 17:55
+
Guozhang Wang 2013-08-20, 18:09
Copy link to this message
-
Re: Avro serialization
This paper has more information on what we are doing at LinkedIn:
http://sites.computer.org/debull/A12june/pipeline.pdf

This Avro JIRA has a schema repository implementation similar to the one
LinkedIn uses:
https://issues.apache.org/jira/browse/AVRO-1124

-Jay
On Tue, Aug 20, 2013 at 7:08 AM, Mark <[EMAIL PROTECTED]> wrote:

> Can someone break down how message serialization would work with Avro?
> I've read instead of adding a schema to every single event it would be wise
> to add some sort of fingerprint with each message to identify which schema
> it should used. What I'm having trouble understanding is, how do we read
> the fingerprint without a schema? Don't we need the schema to deserialize?
>  Same question goes for working with Hadoop.. how does the input format
> know which schema to use?
>
> Thanks

 
+
Mark 2013-08-20, 18:09
+
Mark 2013-08-20, 18:38
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB