Imagine I use Avro to serialize an object (without loss of generality let's
say an array of longs). I'm curious if it is possible to compare those
arrays without deserializing... ie look at the bytes in memory or on disk,
and do the comparison based on those bytes (ie the raw comparison that
Hadoop does in the shuffle sort).

I poked around the documentation but wasn't sure where to look.

Thanks for your help!
Jon

+1

I need this kind of access too, to roll back Avro records that fail to
finish writing when python dies from a UTF error.

Russell Jurney http://datasyndrome.com

On May 22, 2012, at 1:22 PM, Jonathan Coveney <[EMAIL PROTECTED]> wrote:

> Imagine I use Avro to serialize an object (without loss of generality let's say an array of longs). I'm curious if it is possible to compare those arrays without deserializing... ie look at the bytes in memory or on disk, and do the comparison based on those bytes (ie the raw comparison that Hadoop does in the shuffle sort).
>
> I poked around the documentation but wasn't sure where to look.
>
> Thanks for your help!
> Jon

On Tue, May 22, 2012 at 1:22 PM, Jonathan Coveney <[EMAIL PROTECTED]> wrote:
> Imagine I use Avro to serialize an object (without loss of generality let's
> say an array of longs). I'm curious if it is possible to compare those
> arrays without deserializing... ie look at the bytes in memory or on disk,
> and do the comparison based on those bytes (ie the raw comparison that
> Hadoop does in the shuffle sort).
>
> I poked around the documentation but wasn't sure where to look.

Yes, this is possible.

The Java method that does this is BinaryData#compare().

http://avro.apache.org/docs/current/api/java/org/apache/avro/io/BinaryData.html#compare(byte[],
int, byte[], int, org.apache.avro.Schema)

Doug

Thanks Doug, that's perfect!

Sent from my iPhone

On May 23, 2012, at 10:28 AM, Doug Cutting <[EMAIL PROTECTED]> wrote:

> On Tue, May 22, 2012 at 1:22 PM, Jonathan Coveney <[EMAIL PROTECTED]> wrote:
>> Imagine I use Avro to serialize an object (without loss of generality let's
>> say an array of longs). I'm curious if it is possible to compare those
>> arrays without deserializing... ie look at the bytes in memory or on disk,
>> and do the comparison based on those bytes (ie the raw comparison that
>> Hadoop does in the shuffle sort).
>>
>> I poked around the documentation but wasn't sure where to look.
>
> Yes, this is possible.
>
> The Java method that does this is BinaryData#compare().
>
> http://avro.apache.org/docs/current/api/java/org/apache/avro/io/BinaryData.html#compare(byte[],
> int, byte[], int, org.apache.avro.Schema)
>
> Doug