Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Kafka >> mail # user >> Avro serialization


+
Mark 2013-08-20, 14:08
+
Guozhang Wang 2013-08-20, 16:19
+
Mark 2013-08-20, 17:55
Copy link to this message
-
Re: Avro serialization
The magic byte is already in the Kafka Message data format, all you need to
do is upon reading the magic byte for Avro, use your Avro resolver which
treat the first 16 bytes as the schemaId to read the schema.

Guozhang
On Tue, Aug 20, 2013 at 10:54 AM, Mark <[EMAIL PROTECTED]> wrote:

> So basically prepend the information in the header of each message?
> Wouldn't this require changes in all readers/writers?
>
> On Aug 20, 2013, at 9:19 AM, Guozhang Wang <[EMAIL PROTECTED]> wrote:
>
> > Hello Mark,
> >
> > The Kafka message format starts with a magic byte indicating what kind of
> > serialization is used for this message. And if this byte indicates Avro,
> > you can layout your message as starting with the schemaId and then
> followed
> > by message payload. Upon consumption, you can first get the schemaId,
> query
> > Avro for the schema given the id, and then use schema to deserialize the
> > message.
> >
> > Guozhang
> >
> >
> > On Tue, Aug 20, 2013 at 7:08 AM, Mark <[EMAIL PROTECTED]> wrote:
> >
> >> Can someone break down how message serialization would work with Avro?
> >> I've read instead of adding a schema to every single event it would be
> wise
> >> to add some sort of fingerprint with each message to identify which
> schema
> >> it should used. What I'm having trouble understanding is, how do we read
> >> the fingerprint without a schema? Don't we need the schema to
> deserialize?
> >> Same question goes for working with Hadoop.. how does the input format
> >> know which schema to use?
> >>
> >> Thanks
> >
> >
> >
> >
> > --
> > -- Guozhang
>
>
--
-- Guozhang

 
+
Jay Kreps 2013-08-20, 16:33
+
Mark 2013-08-20, 18:09
+
Mark 2013-08-20, 18:38
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB