Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> How to extract byte array from memoryOutputStream


Copy link to this message
-
Re: How to extract byte array from memoryOutputStream
Got it Matt. Thanks !

On Thu, May 17, 2012 at 9:59 PM, Matt Stevenson
<[EMAIL PROTECTED]>wrote:

> But what is the suggested way of transferring encoded bytes over the
>> wire. Avro does not seem to expose the encoded bytes anywhere.
>>
>
> I am using a custom method for transporting the encoding bytes, which is
> why I had to get at the bytes.
> Avro has an HTTP transport mechanisms defined, but I haven't used it and I
> don't know if it is fully implemented in the C++ project.
> For typical Avro usage you should not be accessing the raw bytes, which is
> why nobody bothered to expose them.
>
> So even if I am encoding a small dataset, I would end up
>> reading a min "Chunk_size" of memory block.
>>
>
> The "count" variable should be the exact number of bytes in the stream.
> Both for loops are checking "i<count", and "i" is only incremented in the
> inner loop.  Once you reach the count it will break out of both loops.
>
>
> On Wed, May 16, 2012 at 4:45 AM, Gaurav Nanda <[EMAIL PROTECTED]> wrote:
>
>> Hi Matt, The approach you are using to read the written bytes would
>> end up reading all the bytes allocated but not the bytes actually
>> written. So even if I am encoding a small dataset, I would end up
>> reading a min "Chunk_size" of memory block.
>>
>> On Mon, May 14, 2012 at 9:03 PM, Gaurav Nanda <[EMAIL PROTECTED]>
>> wrote:
>> > Thanks Matt !
>> >
>> > But what is the suggested way of transferring encoded bytes over the
>> > wire. Avro does not seem to expose the encoded bytes anywhere.
>> >
>> > - Gaurav Nanda
>> >
>> > On Sat, May 12, 2012 at 6:29 AM, Matt Stevenson
>> > <[EMAIL PROTECTED]> wrote:
>> >> I made a copy of the MemoryOutputStream with a header, so you can get
>> at the
>> >> bytes.
>> >>
>> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.h
>> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.cpp
>> >>
>> >> Add those to your project.
>> >> include "OpenMemoryOutputStream.h" and call openMemoryOutputStream()
>> rather
>> >> than memoryOutputStream().
>> >>
>> >> The memory is stored in chunks.  Here's an example of iterating
>> through it:
>> >>
>> >>> auto_ptr<OpenMemoryOutputStream> os = openMemoryOutputStream();
>> >>> EncoderPtr e = binaryEncoder();
>> >>> e->init(*os);
>> >>>
>> >>> avro::encode(*e, *t);
>> >>> e->flush();
>> >>>
>> >>> int count = os->byteCount();
>> >>> char* data = new char[count];
>> >>> int i=0;
>> >>> for (std::vector<uint8_t*>::const_iterator it = os->data_.begin(); it
>> !>> >>> os->data_.end() && i<count; ++it) {
>> >>>     uint8_t* chunk = *it;
>> >>>     int size = os->chunkSize_;
>> >>>     for(int j=0; j<size && i<count; j++, i++){
>> >>>         data[i] = chunk[j];
>> >>>     }
>> >>> }
>> >>
>> >>
>> >> On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda <[EMAIL PROTECTED]>
>> wrote:
>> >>>
>> >>> I want to extract byte array from memoryOutputStream to transfer the
>> >>> encoded data. How do I achieve that?
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> Matt Stevenson.
>>
>
>
>
> --
> Matt Stevenson.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB