Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # user - Re: Sending Apache avro serialized data to thrift server


+
Martin Kleppmann 2013-06-17, 05:46
+
Avinash Dongre 2013-06-17, 10:23
Copy link to this message
-
Re: Sending Apache avro serialized data to thrift server
Avinash Dongre 2013-06-18, 01:15
Hi Martin,
Following code works for me.

    int length = input.limit() - input.position();
    byte[] avroSerialized = new byte[length];
    System.arraycopy(input.array(), input.position(), avroSerialized, 0,
length);
thanks
Avinash
On Mon, Jun 17, 2013 at 3:53 PM, Avinash Dongre <[EMAIL PROTECTED]>wrote:

> Thanks Martin,
> With following code exceptions goes away.
>
> DatumReader<GenericRecord> reader = new
> GenericDatumReader<GenericRecord>(schema);
>
>     byte[] avroSerialized = new byte[input.limit() - input.position()];
>     input.put(avroSerialized);
>     BinaryDecoder decoder > DecoderFactory.get().binaryDecoder(avroSerialized, null);
>
>     try {
>       GenericRecord result = reader.read(null, decoder);
>       System.out.println("RESULT : " + result.toString());
>     } catch (IOException e) {
>       // TODO Auto-generated catch block
>       e.printStackTrace();
>     }
>
> But I am not getting any values when I do toString, I will investigate
> further on what is wrong with my code.
>
> Thanks for your quick help.
>
>
> Thanks
> Avinash
>
>
>
> On Mon, Jun 17, 2013 at 11:16 AM, Martin Kleppmann <[EMAIL PROTECTED]>wrote:
>
>> It's hard to tell just by looking at the code -- I suggest you inspect
>> the contents of the array you're decoding in each case, and see if there is
>> any difference.
>>
>> Purely guessing, the ByteBuffer that Thrift gives you may have other data
>> before and after the Avro record, and calling `bf.array()` will give you
>> junk in that case. You may need something more like this (rough outline,
>> please read the API docs):
>>
>> byte[] copy = new byte[bf.limit() - bf.position()];
>> bf.read(copy);
>> DecoderFactory.get().binaryDecoder(copy, null);
>>
>>
>>
>> On 16 June 2013 20:15, Avinash Dongre <[EMAIL PROTECTED]> wrote:
>>
>>>
>>> Hi,
>>>> I am trying to send avro serialized data to my thrift server. But when
>>>> I try to de-serialized I get exception
>>>> org.apache.avro.AvroRuntimeException: Malformed data. Length is
>>>> negative: -51
>>>>
>>>> If I do the same operation in the same vm process everything works fine.
>>>>
>>>> Here is my small code.
>>>>
>>>> public static GenericRecord createContentNestedObject() throws
>>>> Exception {
>>>>     GenericRecord image1 = new GenericData.Record(IMAGE_SCHEMA);
>>>>     image1.put("uri", new Utf8("http://javaone.com/keynote_large.jpg
>>>> "));
>>>>     image1.put("width", 0);
>>>>     image1.put("height", 0);
>>>>     image1.put("size", 2);
>>>>     image1.put("title", new Utf8("Javaone Keynote"));
>>>>     return image1;
>>>>   }
>>>>
>>>>   // Helper Method to serialize the object from avro to bytebuffer
>>>>   public static ByteBuffer serialize(GenericRecord content, Schema
>>>> schema)
>>>>        throws Exception {
>>>>
>>>>     ByteArrayOutputStream out = new ByteArrayOutputStream();
>>>>     DatumWriter<GenericRecord> writer = new
>>>> GenericDatumWriter<GenericRecord>(
>>>>         schema);
>>>>     // create Binary Encoder
>>>>     EncoderFactory ef = new EncoderFactory();
>>>>     BinaryEncoder be = ef.binaryEncoder(out, null);
>>>>     writer.write(content, be);
>>>>      be.flush();
>>>>     out.close();
>>>>
>>>>     return ByteBuffer.wrap(out.toByteArray());
>>>>   }
>>>>
>>>>   public static void main(String[] args) throws Exception {
>>>>     try {
>>>>       GenericRecord rd = createContentNestedObject();
>>>>       bf = serialize(rd, IMAGE_SCHEMA);
>>>>
>>>>     } catch (Exception e) {
>>>>       // TODO Auto-generated catch block
>>>>       e.printStackTrace();
>>>>     }
>>>>
>>>>     // Deserialization.
>>>>     DatumReader<GenericRecord> reader = new
>>>> GenericDatumReader<GenericRecord>(IMAGE_SCHEMA_1);
>>>>     ByteArrayOutputStream out = new ByteArrayOutputStream();
>>>>     try {
>>>>       out.write(bf.array());
>>>>     } catch (IOException e1) {
>>>>       // TODO Auto-generated catch block
>>>>       e1.printStackTrace();
>>>>     }
>>>>     BinaryDecoder decoder >>>> DecoderFactory.get().binaryDecoder(out.toByteArray(), null);