-Re: Avro serialization
Mark 2013-08-20, 17:55
So basically prepend the information in the header of each message? Wouldn't this require changes in all readers/writers?
On Aug 20, 2013, at 9:19 AM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
> Hello Mark,
> The Kafka message format starts with a magic byte indicating what kind of
> serialization is used for this message. And if this byte indicates Avro,
> you can layout your message as starting with the schemaId and then followed
> by message payload. Upon consumption, you can first get the schemaId, query
> Avro for the schema given the id, and then use schema to deserialize the
> On Tue, Aug 20, 2013 at 7:08 AM, Mark <[EMAIL PROTECTED]> wrote:
>> Can someone break down how message serialization would work with Avro?
>> I've read instead of adding a schema to every single event it would be wise
>> to add some sort of fingerprint with each message to identify which schema
>> it should used. What I'm having trouble understanding is, how do we read
>> the fingerprint without a schema? Don't we need the schema to deserialize?
>> Same question goes for working with Hadoop.. how does the input format
>> know which schema to use?
> -- Guozhang