Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> How to use Apache Avro to encode the data in avro binary encoded form?


Copy link to this message
-
Re: How to use Apache Avro to encode the data in avro binary encoded form?
As long as you keep track of the schema somewhere.  You cannot read
the data without the schema used to write it.  So you could store the
schema once for the entire column and directly store the binary
encoding in the cells, but you'll then not be able to alter the schema
without re-writing all of the values.  If you want to be able to alter
the schema then you might prefix each binary-encoded cell value with
an integer schema id, then have a separate table that maps schema ids
to schemas.

Doug

On Fri, Sep 20, 2013 at 2:44 PM, Raihan Jamal <[EMAIL PROTECTED]> wrote:
> Hello Mika/Doug,
>
> I am thinking to write byteData as it is in one of my columns in Cassandra
> Column Family so is this right? I have figured it out how can I store it in
> Cassandra already..
> I just wanted to confirm whether I can write byteData as it is or not?
>
>
>
>
>
> Raihan Jamal
>
>
> On Thu, Sep 19, 2013 at 10:45 AM, raihan26 <[EMAIL PROTECTED]> wrote:
>>
>> Thanks Mika. That worked fine
>>
>> GenericDatumWriter<GenericRecord> writer = new
>> GenericDatumWriter<GenericRecord>(schema);
>>         ByteArrayOutputStream os = new ByteArrayOutputStream();
>>         Encoder e = EncoderFactory.get().binaryEncoder(os, null);
>>         writer.write(record, e);
>>         e.flush();
>>         byte[] byteData = os.toByteArray();
>>         os.close();
>>
>>
>> That means, I can write byteData as the value for one of my columns.
>> Correct? I will figure out whether we can store binary blobs or not in
>> Cassandra. But just to make sure, I need to store byteData as it is as the
>> value for one of my columns.. Right?
>>
>> And also how can I deserialize the above data after I retrieve from
>> Cassandra for that particular column? A simple example will make me
>> understand this as well. Thanks.
>>
>>
>>
>>
>>
>>
>>
>>
>> Raihan Jamal
>>
>>
>> On Thu, Sep 19, 2013 at 1:53 AM, Mika Ristimaki [via Apache Avro] <[hidden
>> email]> wrote:
>>>
>>> Hi Raihan,
>>>
>>> You can try somthing like this, see below.
>>>
>>> On Sep 19, 2013, at 4:04 AM, Raihan Jamal <[hidden email]> wrote:
>>>
>>> > I am trying to use Apache Avro to serialize one of our Attributes data
>>> > and I want to use binary encoding version for that not the JSON version. In
>>> > this case, I don't want to serialize to disk or some avro data files. I am
>>> > planning to write Avro binary encoded-data in Cassandra column family.
>>> >
>>> > Below is the code so far I have- I am not sure how to start encoding
>>> > into binary version.
>>> >
>>> >     Schema schema = new
>>> > Parser().parse((AvroExample.class.getResourceAsStream("/user.avsc")));
>>> >
>>> >     GenericRecord record = new GenericData.Record(schema);
>>> >     record.put("vbs", "99,E,0");
>>> >     record.put("eby", "0,E,,,,,,,15");
>>> >     record.put("ste", "77;2,0;1,16;3");
>>> >
>>> >
>>> >     // what I am supposed to do here to serialize into binary encoded
>>> > version of Avro?
>>> >
>>>         GenericDatumWriter<GenericRecord> writer = new
>>> GenericDatumWriter<GenericRecord>(schema);
>>>         ByteArrayOutputStream os = new ByteArrayOutputStream();
>>>         Encoder e = EncoderFactory.get().binaryEncoder(os, null);
>>>         writer.write(record, e);
>>>         e.flush();
>>>         byte[] byteData = os.toByteArray();
>>>         os.close();
>>> >
>>> >    // then write to Cassandra here.
>>> >
>>>
>>> I don't know anything about cassandra, but I'm pretty sure you can write
>>> binary blobs there.
>>>
>>> -Mika
>>>
>>> >
>>> > Can anyone provide a simple example basis on my code how to do that?
>>> > And then I can use that to write into Cassandra column family.
>>> >
>>> > Thanks.
>>> >
>>> >
>>> >
>>> >
>>> > Raihan Jamal
>>>
>>>
>>>
>>> ________________________________
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>>
>>> http://apache-avro.679487.n3.nabble.com/How-to-use-Apache-Avro-to-encode-the-data-in-avro-binary-encoded-form-tp4028217p4028221.html
>>> To start a new topic under Avro - Users, email [hidden email]