Mark 2013-08-22, 03:15
Neha Narkhede 2013-08-22, 03:38
Neha, thanks for the response.
So the only point of the magic byte is to indicate that the rest of the message is Avro encoded? I noticed that in Camus a 4 byte int id of the schema is written instead of the 16 byte SHA. Is this the new preferred way? Which is compatible with https://issues.apache.org/jira/browse/AVRO-1124?
On Aug 21, 2013, at 8:38 PM, Neha Narkhede <[EMAIL PROTECTED]> wrote:
> We define the LinkedIn Kafka message to have a magic byte (indicating Avro
> serialization), MD5 header followed by the payload. The Hadoop consumer
> reads the MD5, looks up the schema in the repository and deserializes the
> On Wed, Aug 21, 2013 at 8:15 PM, Mark <[EMAIL PROTECTED]> wrote:
>> Does LinkedIn include the SHA of the schema into the header of each Avro
>> message they write or do they wrap the avro message and prepend the SHA?
>> In either case, how does the Hadoop consumer know what schema to read?
Neha Narkhede 2013-08-22, 16:37
Mark 2013-08-22, 16:49
Mark 2013-08-22, 16:52