Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Kafka >> mail # user >> More questions on avro serialization

Copy link to this message
Re: More questions on avro serialization
We define the LinkedIn Kafka message to have a magic byte (indicating Avro
serialization), MD5 header followed by the payload. The Hadoop consumer
reads the MD5, looks up the schema in the repository and deserializes the

On Wed, Aug 21, 2013 at 8:15 PM, Mark <[EMAIL PROTECTED]> wrote:

> Does LinkedIn include the SHA of the schema into the header of each Avro
> message they write or do they wrap the avro message and prepend the SHA?
> In either case, how does the Hadoop consumer know what schema to read?