Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Avro serialization

Mark 2013-08-20, 14:08
Copy link to this message
Re: Avro serialization
Hello Mark,

The Kafka message format starts with a magic byte indicating what kind of
serialization is used for this message. And if this byte indicates Avro,
you can layout your message as starting with the schemaId and then followed
by message payload. Upon consumption, you can first get the schemaId, query
Avro for the schema given the id, and then use schema to deserialize the

On Tue, Aug 20, 2013 at 7:08 AM, Mark <[EMAIL PROTECTED]> wrote:

> Can someone break down how message serialization would work with Avro?
> I've read instead of adding a schema to every single event it would be wise
> to add some sort of fingerprint with each message to identify which schema
> it should used. What I'm having trouble understanding is, how do we read
> the fingerprint without a schema? Don't we need the schema to deserialize?
>  Same question goes for working with Hadoop.. how does the input format
> know which schema to use?
> Thanks
-- Guozhang

Mark 2013-08-20, 17:55
Guozhang Wang 2013-08-20, 18:09
Jay Kreps 2013-08-20, 16:33
Mark 2013-08-20, 18:09
Mark 2013-08-20, 18:38