Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Secondary sort in hadoop with avro

Frank Kootte 2012-09-11, 15:36
Copy link to this message
RE: Secondary sort in hadoop with avro

I have spent a bit of time doing this recently but with MR2 and CDH4 which may not be appropriate to your use case. However assuming some similarities, I suspect your problem is that you also need to override compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) on AvroKeyComparator.
The advantage to Avro is that Hadoop does not need to deserialize to sort in the shuffle. This function in RawComparator allows Hadoop to quickly compare the bytes directly.
Whilst this seems a bit daunting my trick to doing this in MR2 is to leverage Avro's excellent support for projections - subsets of schemas. For example let's say you want to "group" by attribute A but then "sort" by attribute B. In this case I would use a composite key with schema {A, B} and the out of the box AvroKeyComparator as the sort comparator. Then I would implement my own grouping comparator which uses a schema of just {A} then uses the BinaryData function to compare:
I assume you can do something similar in MR1.

> Subject: Secondary sort in hadoop with avro
> Date: Tue, 11 Sep 2012 17:36:06 +0200
> I need to implement secondary sort within an avro based MR sequence. I however find little to documentation or examples online.
> I would like to implement this by overriding the  'int compare(AvroWrapper<T> x, AvroWrapper<T> y)' method but I fail to have it invoked.
> Does anybody have experience implementing secondary sort on deserialised avro objects ?
> Some help, advise or pointers will be very much appreciated !
Frank Kootte 2012-09-12, 06:51
Frank Kootte 2012-09-12, 14:42
Jacob Metcalf 2012-09-13, 09:24