Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - Apache Avro Encoding types?


Copy link to this message
-
Re: Apache Avro Encoding types?
Raihan Jamal 2013-09-19, 03:39
Thanks Connor. That makes sense... That whats we are planning to do. We
will have Cassandra column which will have composite columns. And those
composite column will have-

Avro-Binary-Encoded_value   Avro-Schema-Name-For-This particular-column
Last-Modified-Date

So when we will retrieve the data for a particular USER-ID for the above
column, we will know its Schema-Name, then we can pull the actual schema
basis on this schema-name, and then we can deserialize the
Avro-Binary-Encoded_value using that schema.
*Raihan Jamal*
On Wed, Sep 18, 2013 at 7:15 PM, Connor Doyle <[EMAIL PROTECTED]> wrote:

> Hi Raihan,
>
> One strategy may be to store a cache of schemata elsewhere and then write
> just enough information (version number, md5, or similar) to look up the
> right schema version adjacent to your avro/binary blob column.  This would
> impose little overhead and still allow you to take advantage of Avro's
> built-in schema resolution to read old serialized data.
> --
> Connor
>
> On Sep 18, 2013, at 20:03, raihan26 <[EMAIL PROTECTED]> wrote:
>
> Thanks a lot Doug. So that means, If I need to store avro binary encoded
> data in Cassandra column family, then the actual schema won't get stored
> with that? Right? I need to have the schema through some external means
> whenever I am reading that binary encoded data. Right?
>
> *Raihan Jamal*
>
>
> On Wed, Sep 18, 2013 at 2:28 PM, Doug Cutting [via Apache Avro] <[hidden
> email] <http://user/SendEmail.jtp?type=node&node=4028216&i=0>> wrote:
>
>> On Wed, Sep 18, 2013 at 1:53 PM, Raihan Jamal <[hidden email]<http://user/SendEmail.jtp?type=node&node=4028215&i=0>>
>> wrote:
>> > Suppose if I am going with Binary encoding route instead of JSON then
>> does
>> > the schema also gets stored in that binary encoding format always?
>>
>> The schema is stored in Avro data files.  Avro RPC also manages
>> schemas for you.  But if you store Avro binary-encoded data in some
>> other container then you may need to keep track of the schema
>> yourself.
>>
>> Doug
>>
>>
>> ------------------------------
>>  If you reply to this email, your message will be added to the
>> discussion below:
>>
>> http://apache-avro.679487.n3.nabble.com/Apache-Avro-Encoding-types-tp4028194p4028215.html
>>  To start a new topic under Avro - Users, email [hidden email]<http://user/SendEmail.jtp?type=node&node=4028216&i=1>
>> To unsubscribe from Apache Avro, click here.
>> NAML<http://apache-avro.679487.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
> ------------------------------
> View this message in context: Re: Apache Avro Encoding types?<http://apache-avro.679487.n3.nabble.com/Apache-Avro-Encoding-types-tp4028194p4028216.html>
> Sent from the Avro - Users mailing list archive<http://apache-avro.679487.n3.nabble.com/Avro-Users-f679479.html>at
> Nabble.com.
>
>