Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Re: Sending Apache avro serialized data to thrift server


Copy link to this message
-
Re: Sending Apache avro serialized data to thrift server
It's hard to tell just by looking at the code -- I suggest you inspect the
contents of the array you're decoding in each case, and see if there is any
difference.

Purely guessing, the ByteBuffer that Thrift gives you may have other data
before and after the Avro record, and calling `bf.array()` will give you
junk in that case. You may need something more like this (rough outline,
please read the API docs):

byte[] copy = new byte[bf.limit() - bf.position()];
bf.read(copy);
DecoderFactory.get().binaryDecoder(copy, null);

On 16 June 2013 20:15, Avinash Dongre <[EMAIL PROTECTED]> wrote:

>
> Hi,
>> I am trying to send avro serialized data to my thrift server. But when I
>> try to de-serialized I get exception
>> org.apache.avro.AvroRuntimeException: Malformed data. Length is negative:
>> -51
>>
>> If I do the same operation in the same vm process everything works fine.
>>
>> Here is my small code.
>>
>> public static GenericRecord createContentNestedObject() throws Exception {
>>     GenericRecord image1 = new GenericData.Record(IMAGE_SCHEMA);
>>     image1.put("uri", new Utf8("http://javaone.com/keynote_large.jpg"));
>>     image1.put("width", 0);
>>     image1.put("height", 0);
>>     image1.put("size", 2);
>>     image1.put("title", new Utf8("Javaone Keynote"));
>>     return image1;
>>   }
>>
>>   // Helper Method to serialize the object from avro to bytebuffer
>>   public static ByteBuffer serialize(GenericRecord content, Schema schema)
>>        throws Exception {
>>
>>     ByteArrayOutputStream out = new ByteArrayOutputStream();
>>     DatumWriter<GenericRecord> writer = new
>> GenericDatumWriter<GenericRecord>(
>>         schema);
>>     // create Binary Encoder
>>     EncoderFactory ef = new EncoderFactory();
>>     BinaryEncoder be = ef.binaryEncoder(out, null);
>>     writer.write(content, be);
>>      be.flush();
>>     out.close();
>>
>>     return ByteBuffer.wrap(out.toByteArray());
>>   }
>>
>>   public static void main(String[] args) throws Exception {
>>     try {
>>       GenericRecord rd = createContentNestedObject();
>>       bf = serialize(rd, IMAGE_SCHEMA);
>>
>>     } catch (Exception e) {
>>       // TODO Auto-generated catch block
>>       e.printStackTrace();
>>     }
>>
>>     // Deserialization.
>>     DatumReader<GenericRecord> reader = new
>> GenericDatumReader<GenericRecord>(IMAGE_SCHEMA_1);
>>     ByteArrayOutputStream out = new ByteArrayOutputStream();
>>     try {
>>       out.write(bf.array());
>>     } catch (IOException e1) {
>>       // TODO Auto-generated catch block
>>       e1.printStackTrace();
>>     }
>>     BinaryDecoder decoder >> DecoderFactory.get().binaryDecoder(out.toByteArray(), null);
>>     try {
>>       GenericRecord result = reader.read(null, decoder);
>>       System.out.println("RESULT : " + result.toString());
>>     } catch (IOException e) {
>>       // TODO Auto-generated catch block
>>       e.printStackTrace();
>>     }
>>
>> This code works. When I send the bf to my thrift api , I am getting the
>> error,
>> my thrift api is simple which accept ByteBuffer and try to deserialize it.
>>
>> following is the code in the thrift server api
>>
>> DatumReader<GenericRecord> reader = new
>> GenericDatumReader<GenericRecord>(IMAGE_SCHEMA_1);
>>     ByteArrayOutputStream out = new ByteArrayOutputStream();
>>     try {
>>       out.write(bf.array());
>>     } catch (IOException e1) {
>>       // TODO Auto-generated catch block
>>       e1.printStackTrace();
>>     }
>>     BinaryDecoder decoder >> DecoderFactory.get().binaryDecoder(out.toByteArray(), null);
>>     try {
>>       GenericRecord result = reader.read(null, decoder);
>>       System.out.println("RESULT : " + result.toString());
>>     } catch (IOException e) {
>>       // TODO Auto-generated catch block
>>       e.printStackTrace();
>>     }
>>
>>
>>
>> What is wrong here.
>>
>>
>> Thanks
>> Aviansh
>>
>>
>>
>
+
Avinash Dongre 2013-06-17, 10:23
+
Avinash Dongre 2013-06-18, 01:15