Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Avro serialization


+
Mark 2013-08-20, 14:08
+
Guozhang Wang 2013-08-20, 16:19
Copy link to this message
-
Re: Avro serialization
So basically prepend the information in the header of each message? Wouldn't this require changes in all readers/writers?

On Aug 20, 2013, at 9:19 AM, Guozhang Wang <[EMAIL PROTECTED]> wrote:

> Hello Mark,
>
> The Kafka message format starts with a magic byte indicating what kind of
> serialization is used for this message. And if this byte indicates Avro,
> you can layout your message as starting with the schemaId and then followed
> by message payload. Upon consumption, you can first get the schemaId, query
> Avro for the schema given the id, and then use schema to deserialize the
> message.
>
> Guozhang
>
>
> On Tue, Aug 20, 2013 at 7:08 AM, Mark <[EMAIL PROTECTED]> wrote:
>
>> Can someone break down how message serialization would work with Avro?
>> I've read instead of adding a schema to every single event it would be wise
>> to add some sort of fingerprint with each message to identify which schema
>> it should used. What I'm having trouble understanding is, how do we read
>> the fingerprint without a schema? Don't we need the schema to deserialize?
>> Same question goes for working with Hadoop.. how does the input format
>> know which schema to use?
>>
>> Thanks
>
>
>
>
> --
> -- Guozhang
 
+
Guozhang Wang 2013-08-20, 18:09
+
Jay Kreps 2013-08-20, 16:33
+
Mark 2013-08-20, 18:09
+
Mark 2013-08-20, 18:38