Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Re: Sending Apache avro serialized data to thrift server


Copy link to this message
-
Re: Sending Apache avro serialized data to thrift server
Thanks Martin,
With following code exceptions goes away.

DatumReader<GenericRecord> reader = new
GenericDatumReader<GenericRecord>(schema);

    byte[] avroSerialized = new byte[input.limit() - input.position()];
    input.put(avroSerialized);
    BinaryDecoder decoder DecoderFactory.get().binaryDecoder(avroSerialized, null);

    try {
      GenericRecord result = reader.read(null, decoder);
      System.out.println("RESULT : " + result.toString());
    } catch (IOException e) {
      // TODO Auto-generated catch block
      e.printStackTrace();
    }

But I am not getting any values when I do toString, I will investigate
further on what is wrong with my code.

Thanks for your quick help.
Thanks
Avinash

On Mon, Jun 17, 2013 at 11:16 AM, Martin Kleppmann <[EMAIL PROTECTED]>wrote:

> It's hard to tell just by looking at the code -- I suggest you inspect the
> contents of the array you're decoding in each case, and see if there is any
> difference.
>
> Purely guessing, the ByteBuffer that Thrift gives you may have other data
> before and after the Avro record, and calling `bf.array()` will give you
> junk in that case. You may need something more like this (rough outline,
> please read the API docs):
>
> byte[] copy = new byte[bf.limit() - bf.position()];
> bf.read(copy);
> DecoderFactory.get().binaryDecoder(copy, null);
>
>
>
> On 16 June 2013 20:15, Avinash Dongre <[EMAIL PROTECTED]> wrote:
>
>>
>> Hi,
>>> I am trying to send avro serialized data to my thrift server. But when I
>>> try to de-serialized I get exception
>>> org.apache.avro.AvroRuntimeException: Malformed data. Length is
>>> negative: -51
>>>
>>> If I do the same operation in the same vm process everything works fine.
>>>
>>> Here is my small code.
>>>
>>> public static GenericRecord createContentNestedObject() throws Exception
>>> {
>>>     GenericRecord image1 = new GenericData.Record(IMAGE_SCHEMA);
>>>     image1.put("uri", new Utf8("http://javaone.com/keynote_large.jpg"));
>>>     image1.put("width", 0);
>>>     image1.put("height", 0);
>>>     image1.put("size", 2);
>>>     image1.put("title", new Utf8("Javaone Keynote"));
>>>     return image1;
>>>   }
>>>
>>>   // Helper Method to serialize the object from avro to bytebuffer
>>>   public static ByteBuffer serialize(GenericRecord content, Schema
>>> schema)
>>>        throws Exception {
>>>
>>>     ByteArrayOutputStream out = new ByteArrayOutputStream();
>>>     DatumWriter<GenericRecord> writer = new
>>> GenericDatumWriter<GenericRecord>(
>>>         schema);
>>>     // create Binary Encoder
>>>     EncoderFactory ef = new EncoderFactory();
>>>     BinaryEncoder be = ef.binaryEncoder(out, null);
>>>     writer.write(content, be);
>>>      be.flush();
>>>     out.close();
>>>
>>>     return ByteBuffer.wrap(out.toByteArray());
>>>   }
>>>
>>>   public static void main(String[] args) throws Exception {
>>>     try {
>>>       GenericRecord rd = createContentNestedObject();
>>>       bf = serialize(rd, IMAGE_SCHEMA);
>>>
>>>     } catch (Exception e) {
>>>       // TODO Auto-generated catch block
>>>       e.printStackTrace();
>>>     }
>>>
>>>     // Deserialization.
>>>     DatumReader<GenericRecord> reader = new
>>> GenericDatumReader<GenericRecord>(IMAGE_SCHEMA_1);
>>>     ByteArrayOutputStream out = new ByteArrayOutputStream();
>>>     try {
>>>       out.write(bf.array());
>>>     } catch (IOException e1) {
>>>       // TODO Auto-generated catch block
>>>       e1.printStackTrace();
>>>     }
>>>     BinaryDecoder decoder >>> DecoderFactory.get().binaryDecoder(out.toByteArray(), null);
>>>     try {
>>>       GenericRecord result = reader.read(null, decoder);
>>>       System.out.println("RESULT : " + result.toString());
>>>     } catch (IOException e) {
>>>       // TODO Auto-generated catch block
>>>       e.printStackTrace();
>>>     }
>>>
>>> This code works. When I send the bf to my thrift api , I am getting the
>>> error,
>>> my thrift api is simple which accept ByteBuffer and try to deserialize
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB