Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro, mail # user - Secondary Sort Helper


Copy link to this message
-
Secondary Sort Helper
Johannes Schulte 2013-05-17, 20:22
Hi all,

I am using a lot of secondary sort Comparators in my avro map reduce jobs.
However I haven't found a comfortable way of constructing the grouping
schema (sorting schema is mostly the default) or the partitioner for the
binary data compare method in AvroKeyComparator.

So in my jobs i mostly construct a manual schema with Schema.createRecord()
but that is really verbose. I also have some jobs where i created new
schema definitions with the correct grouping order but that is tedious
either.

Does anyone have an idea of how to make this easier? I could think of
either some meta annotations
in the schema or some property based reflection stuff. In the end it's
mostly one field from the record that's partitoned on and one or two that
are used as grouping comparators.

I think the basic toolset for doing this should be there..I just need some
hints..

Cheers,

Johannes