Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Custom Partitioner is now working in CDH4


Copy link to this message
-
Custom Partitioner is now working in CDH4
Hi  All
   Just we migrated from CDH3 to CDH4 , But after that custom Partitioner
is note working under CDH4.

Can you tell me why it is like that but in CDH3  same code is working fine.

PARTITONER CODE
-------------------------------------------
public static class AgePartitioner extends Partitioner<Text, Text> {
        static{

System.out.println("&##############################AgePartitioner++++++++++++++++++++++");
        }
        @Override
        public int getPartition(Text key, Text value, int numReduceTasks) {

System.out.println("&##############################AgePartitioner"+value.toString());
            String [] nameAgeScore = value.toString().split("\t");
            String age = nameAgeScore[1];
            int ageInt = Integer.parseInt(age);

            //this is done to avoid performing mod with 0
            if(numReduceTasks == 0)
                return 0;

            //if the age is <20, assign partition 0
            if(ageInt <=20){
                return 0;
            }
            //else if the age is between 20 and 50, assign partition 1
            if(ageInt >20 && ageInt <=50){

                return 1 % numReduceTasks;
            }
            //otherwise assign partition 2
            else
                return 2 % numReduceTasks;

        }
    }

DRIVER LEVEL CODE
-------------------------------------------

job.setPartitionerClass(AgePartitioner.class);
job.setNumReduceTasks(3);
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB