Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> More questions on avro serialization

Copy link to this message
Re: More questions on avro serialization
Neha, thanks for the response.

So the only point of the magic byte is to indicate that the rest of the message is Avro encoded? I noticed that in Camus a 4 byte int id of the schema is written instead of the 16 byte SHA. Is this the new preferred way? Which is compatible with https://issues.apache.org/jira/browse/AVRO-1124?

Thanks again

On Aug 21, 2013, at 8:38 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote:

> We define the LinkedIn Kafka message to have a magic byte (indicating Avro
> serialization), MD5 header followed by the payload. The Hadoop consumer
> reads the MD5, looks up the schema in the repository and deserializes the
> message.
> Thanks,
> Neha
> On Wed, Aug 21, 2013 at 8:15 PM, Mark <[EMAIL PROTECTED]> wrote:
>> Does LinkedIn include the SHA of the schema into the header of each Avro
>> message they write or do they wrap the avro message and prepend the SHA?
>> In either case, how does the Hadoop consumer know what schema to read?