Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro, mail # user - Setting bytes in Java


Copy link to this message
-
Re: Setting bytes in Java
Scott Carey 2011-01-18, 18:30
Please open a bug report in JIRA.  I don't have time to look at this now,
but someone else might.
On the topic of per record versioning and how to design a system that does
not store schemas per record, there have been useful topics on this
mailing list in the past:
http://search-hadoop.com/m/66jvQoopYw/HAvroBase&subj=Re+question+about+comp
letely+untagged+data+

http://search-hadoop.com/m/q7lLU1GVhHd2/HAvroBase&subj=Re+Versioning+of+an+
array+of+a+record

On 1/18/11 10:08 AM, "David Rosenstrauch" <[EMAIL PROTECTED]> wrote:

>I've also found this to be the case, and was wondering about it.  I also
>had thought that I could just re-init an existing BinaryEncoder, but
>found that I had to create a new one each time.  I didn't really think
>much of it at the time, but in retrospect it does sound like it might be
>a bug.  Perhaps one of the devs can comment more.  (And/or perhaps you
>might want to open a bug report about this.)
>
>DR
>
>On 01/18/2011 03:17 AM, Devajyoti Sarkar wrote:
>> Let me first give some context, I would like to store a datum serialized
>> with a BinaryEncoder without having to place a schema with it (as the
>> DataFileWriter does). Instead I have created a container record that
>>stores
>> a unique id for the schema version and a payload field of type "bytes".
>>This
>> allows me to have a self-describing data object (for example, to place
>>in a
>> cell in HBase) without the overhead of a schema per object. (Perhaps
>>there
>> is a better way to do this, if so please let me know).
>>
>> The code looks something like this:
>>
>>      GenericRecord container = new GenericData.Record(containerSchema);
>>      writer.setSchema(containerSchema);
>>      container.put(CONTAINER_SCHEMA_ID_FIELD,
>> datum.getSchema().getProp(SCHEMA_ID_PROPERTY));
>>      container.put(CONTAINER_PAYLOAD_FIELD,
>> ByteBuffer.wrap(datumBits.toByteArray()));
>>      ByteArrayOutputStream containerBits = new ByteArrayOutputStream();
>>      encoder.init(containerBits);
>>      writer.write(container, encoder);
>>      encoder.flush();
>>      containerBits.flush();
>>      containerBits.close();
>>
>> I am trying to reuse an encoder by calling init() to re-initialize it.
>> Perhaps this is what creates the problem. If I create a new encoder each
>> time everything works fine. However, if I just use init, then the
>> OutputStream for the encoder is reset but the OutputStream for the
>> SimpleByteWriter within the encoder is not. This seems to be causing the
>> problem because when the encoder is flushed, it does not write the
>>bytes in
>> the ByteWriter. Perhaps the init() method is not supposed to be used
>>this
>> way. But it would be nice to not have to create a new encoder each time.
>>
>> Can you please let me know if the above looks right and advise me as to
>>what
>> is the best way to do the serialization.
>>
>> Thanks,
>> Dev
>>
>>
>>
>> On Tue, Jan 18, 2011 at 4:14 AM, Scott
>>Carey<[EMAIL PROTECTED]>wrote:
>>
>>> BinaryEncoder buffers data, you may have to call flush() to see it in
>>>the
>>> output stream.
>>>
>>>
>>> On 1/17/11 4:53 AM, "Devajyoti Sarkar"<[EMAIL PROTECTED]>  wrote:
>>>
>>> Hi,
>>>
>>> I am just beginning to use Avro, so I apologize if this is a silly
>>> question.
>>>
>>> I would like to set a field of type "bytes" in Java. I am assuming
>>>that all
>>> I need to do is wrap a byte[] in a ByteBuffer to set the value.
>>> Unfortunately that does not seem to work. I am using a BinaryEncoder
>>>and
>>> looking at its output, it has not written any the bytes that were in
>>>the
>>> array. The first four values of the array are 0, -128, -128, -128.
>>>
>>> Is it because Java uses 8-bit signed bytes while the Avro spec calls
>>>for
>>> 8-bit unsigned bytes in a field of type "bytes"? If so, how does one
>>>convert
>>> Java bytes to the kind accepted by Avro?
>>>
>>> Thanks in advance.
>>>
>>> Dev
>>>
>>>
>>
>