Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> How to extract byte array from memoryOutputStream


Copy link to this message
-
Re: How to extract byte array from memoryOutputStream
Hi Matt, The approach you are using to read the written bytes would
end up reading all the bytes allocated but not the bytes actually
written. So even if I am encoding a small dataset, I would end up
reading a min "Chunk_size" of memory block.

On Mon, May 14, 2012 at 9:03 PM, Gaurav Nanda <[EMAIL PROTECTED]> wrote:
> Thanks Matt !
>
> But what is the suggested way of transferring encoded bytes over the
> wire. Avro does not seem to expose the encoded bytes anywhere.
>
> - Gaurav Nanda
>
> On Sat, May 12, 2012 at 6:29 AM, Matt Stevenson
> <[EMAIL PROTECTED]> wrote:
>> I made a copy of the MemoryOutputStream with a header, so you can get at the
>> bytes.
>>
>> http://branchingworlds.com/avro/OpenMemoryOutputStream.h
>> http://branchingworlds.com/avro/OpenMemoryOutputStream.cpp
>>
>> Add those to your project.
>> include "OpenMemoryOutputStream.h" and call openMemoryOutputStream() rather
>> than memoryOutputStream().
>>
>> The memory is stored in chunks.  Here's an example of iterating through it:
>>
>>> auto_ptr<OpenMemoryOutputStream> os = openMemoryOutputStream();
>>> EncoderPtr e = binaryEncoder();
>>> e->init(*os);
>>>
>>> avro::encode(*e, *t);
>>> e->flush();
>>>
>>> int count = os->byteCount();
>>> char* data = new char[count];
>>> int i=0;
>>> for (std::vector<uint8_t*>::const_iterator it = os->data_.begin(); it !>>> os->data_.end() && i<count; ++it) {
>>>     uint8_t* chunk = *it;
>>>     int size = os->chunkSize_;
>>>     for(int j=0; j<size && i<count; j++, i++){
>>>         data[i] = chunk[j];
>>>     }
>>> }
>>
>>
>> On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda <[EMAIL PROTECTED]> wrote:
>>>
>>> I want to extract byte array from memoryOutputStream to transfer the
>>> encoded data. How do I achieve that?
>>
>>
>>
>>
>> --
>> Matt Stevenson.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB