Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> How to extract byte array from memoryOutputStream


Copy link to this message
-
Re: How to extract byte array from memoryOutputStream
Got it Matt. Thanks !

On Thu, May 17, 2012 at 9:59 PM, Matt Stevenson
<[EMAIL PROTECTED]>wrote:

> But what is the suggested way of transferring encoded bytes over the
>> wire. Avro does not seem to expose the encoded bytes anywhere.
>>
>
> I am using a custom method for transporting the encoding bytes, which is
> why I had to get at the bytes.
> Avro has an HTTP transport mechanisms defined, but I haven't used it and I
> don't know if it is fully implemented in the C++ project.
> For typical Avro usage you should not be accessing the raw bytes, which is
> why nobody bothered to expose them.
>
> So even if I am encoding a small dataset, I would end up
>> reading a min "Chunk_size" of memory block.
>>
>
> The "count" variable should be the exact number of bytes in the stream.
> Both for loops are checking "i<count", and "i" is only incremented in the
> inner loop.  Once you reach the count it will break out of both loops.
>
>
> On Wed, May 16, 2012 at 4:45 AM, Gaurav Nanda <[EMAIL PROTECTED]> wrote:
>
>> Hi Matt, The approach you are using to read the written bytes would
>> end up reading all the bytes allocated but not the bytes actually
>> written. So even if I am encoding a small dataset, I would end up
>> reading a min "Chunk_size" of memory block.
>>
>> On Mon, May 14, 2012 at 9:03 PM, Gaurav Nanda <[EMAIL PROTECTED]>
>> wrote:
>> > Thanks Matt !
>> >
>> > But what is the suggested way of transferring encoded bytes over the
>> > wire. Avro does not seem to expose the encoded bytes anywhere.
>> >
>> > - Gaurav Nanda
>> >
>> > On Sat, May 12, 2012 at 6:29 AM, Matt Stevenson
>> > <[EMAIL PROTECTED]> wrote:
>> >> I made a copy of the MemoryOutputStream with a header, so you can get
>> at the
>> >> bytes.
>> >>
>> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.h
>> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.cpp
>> >>
>> >> Add those to your project.
>> >> include "OpenMemoryOutputStream.h" and call openMemoryOutputStream()
>> rather
>> >> than memoryOutputStream().
>> >>
>> >> The memory is stored in chunks.  Here's an example of iterating
>> through it:
>> >>
>> >>> auto_ptr<OpenMemoryOutputStream> os = openMemoryOutputStream();
>> >>> EncoderPtr e = binaryEncoder();
>> >>> e->init(*os);
>> >>>
>> >>> avro::encode(*e, *t);
>> >>> e->flush();
>> >>>
>> >>> int count = os->byteCount();
>> >>> char* data = new char[count];
>> >>> int i=0;
>> >>> for (std::vector<uint8_t*>::const_iterator it = os->data_.begin(); it
>> !>> >>> os->data_.end() && i<count; ++it) {
>> >>>     uint8_t* chunk = *it;
>> >>>     int size = os->chunkSize_;
>> >>>     for(int j=0; j<size && i<count; j++, i++){
>> >>>         data[i] = chunk[j];
>> >>>     }
>> >>> }
>> >>
>> >>
>> >> On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda <[EMAIL PROTECTED]>
>> wrote:
>> >>>
>> >>> I want to extract byte array from memoryOutputStream to transfer the
>> >>> encoded data. How do I achieve that?
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Matt Stevenson.
>>
>
>
>
> --
> Matt Stevenson.
>