Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> More questions on avro serialization


Copy link to this message
-
Re: More questions on avro serialization
Neha, thanks for the response.

So the only point of the magic byte is to indicate that the rest of the message is Avro encoded? I noticed that in Camus a 4 byte int id of the schema is written instead of the 16 byte SHA. Is this the new preferred way? Which is compatible with https://issues.apache.org/jira/browse/AVRO-1124?

Thanks again

On Aug 21, 2013, at 8:38 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote:

> We define the LinkedIn Kafka message to have a magic byte (indicating Avro
> serialization), MD5 header followed by the payload. The Hadoop consumer
> reads the MD5, looks up the schema in the repository and deserializes the
> message.
>
> Thanks,
> Neha
>
>
> On Wed, Aug 21, 2013 at 8:15 PM, Mark <[EMAIL PROTECTED]> wrote:
>
>> Does LinkedIn include the SHA of the schema into the header of each Avro
>> message they write or do they wrap the avro message and prepend the SHA?
>>
>> In either case, how does the Hadoop consumer know what schema to read?
 
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB