Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # user - How to extract byte array from memoryOutputStream


+
Gaurav Nanda 2012-05-10, 17:19
+
Matt Stevenson 2012-05-12, 00:59
+
Gaurav Nanda 2012-05-14, 15:33
Copy link to this message
-
Re: How to extract byte array from memoryOutputStream
Gaurav Nanda 2012-05-16, 09:45
Hi Matt, The approach you are using to read the written bytes would
end up reading all the bytes allocated but not the bytes actually
written. So even if I am encoding a small dataset, I would end up
reading a min "Chunk_size" of memory block.

On Mon, May 14, 2012 at 9:03 PM, Gaurav Nanda <[EMAIL PROTECTED]> wrote:
> Thanks Matt !
>
> But what is the suggested way of transferring encoded bytes over the
> wire. Avro does not seem to expose the encoded bytes anywhere.
>
> - Gaurav Nanda
>
> On Sat, May 12, 2012 at 6:29 AM, Matt Stevenson
> <[EMAIL PROTECTED]> wrote:
>> I made a copy of the MemoryOutputStream with a header, so you can get at the
>> bytes.
>>
>> http://branchingworlds.com/avro/OpenMemoryOutputStream.h
>> http://branchingworlds.com/avro/OpenMemoryOutputStream.cpp
>>
>> Add those to your project.
>> include "OpenMemoryOutputStream.h" and call openMemoryOutputStream() rather
>> than memoryOutputStream().
>>
>> The memory is stored in chunks.  Here's an example of iterating through it:
>>
>>> auto_ptr<OpenMemoryOutputStream> os = openMemoryOutputStream();
>>> EncoderPtr e = binaryEncoder();
>>> e->init(*os);
>>>
>>> avro::encode(*e, *t);
>>> e->flush();
>>>
>>> int count = os->byteCount();
>>> char* data = new char[count];
>>> int i=0;
>>> for (std::vector<uint8_t*>::const_iterator it = os->data_.begin(); it !>>> os->data_.end() && i<count; ++it) {
>>>     uint8_t* chunk = *it;
>>>     int size = os->chunkSize_;
>>>     for(int j=0; j<size && i<count; j++, i++){
>>>         data[i] = chunk[j];
>>>     }
>>> }
>>
>>
>> On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda <[EMAIL PROTECTED]> wrote:
>>>
>>> I want to extract byte array from memoryOutputStream to transfer the
>>> encoded data. How do I achieve that?
>>
>>
>>
>>
>> --
>> Matt Stevenson.
+
Matt Stevenson 2012-05-17, 16:29
+
Gaurav Nanda 2012-05-17, 16:35