Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka, mail # user - More questions on avro serialization


+
Mark 2013-08-22, 03:15
+
Neha Narkhede 2013-08-22, 03:38
Copy link to this message
-
Re: More questions on avro serialization
Mark 2013-08-22, 04:01
Neha, thanks for the response.

So the only point of the magic byte is to indicate that the rest of the message is Avro encoded? I noticed that in Camus a 4 byte int id of the schema is written instead of the 16 byte SHA. Is this the new preferred way? Which is compatible with https://issues.apache.org/jira/browse/AVRO-1124?

Thanks again

On Aug 21, 2013, at 8:38 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote:

> We define the LinkedIn Kafka message to have a magic byte (indicating Avro
> serialization), MD5 header followed by the payload. The Hadoop consumer
> reads the MD5, looks up the schema in the repository and deserializes the
> message.
>
> Thanks,
> Neha
>
>
> On Wed, Aug 21, 2013 at 8:15 PM, Mark <[EMAIL PROTECTED]> wrote:
>
>> Does LinkedIn include the SHA of the schema into the header of each Avro
>> message they write or do they wrap the avro message and prepend the SHA?
>>
>> In either case, how does the Hadoop consumer know what schema to read?
 
+
Neha Narkhede 2013-08-22, 16:37
+
Mark 2013-08-22, 16:49
+
Mark 2013-08-22, 16:52