Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop, mail # user - Custom Partitioner is now working in CDH4


Copy link to this message
-
Custom Partitioner is now working in CDH4
samir das mohapatra 2013-01-23, 16:58
Hi  All
   Just we migrated from CDH3 to CDH4 , But after that custom Partitioner
is note working under CDH4.

Can you tell me why it is like that but in CDH3  same code is working fine.

PARTITONER CODE
-------------------------------------------
public static class AgePartitioner extends Partitioner<Text, Text> {
        static{

System.out.println("&##############################AgePartitioner++++++++++++++++++++++");
        }
        @Override
        public int getPartition(Text key, Text value, int numReduceTasks) {

System.out.println("&##############################AgePartitioner"+value.toString());
            String [] nameAgeScore = value.toString().split("\t");
            String age = nameAgeScore[1];
            int ageInt = Integer.parseInt(age);

            //this is done to avoid performing mod with 0
            if(numReduceTasks == 0)
                return 0;

            //if the age is <20, assign partition 0
            if(ageInt <=20){
                return 0;
            }
            //else if the age is between 20 and 50, assign partition 1
            if(ageInt >20 && ageInt <=50){

                return 1 % numReduceTasks;
            }
            //otherwise assign partition 2
            else
                return 2 % numReduceTasks;

        }
    }

DRIVER LEVEL CODE
-------------------------------------------

job.setPartitionerClass(AgePartitioner.class);
job.setNumReduceTasks(3);