Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Re: Custom Partitioner is now working in CDH4


Copy link to this message
-
Re: Custom Partitioner is now working in CDH4
Moving to cdh-user@ as you seem to indicate this is CDH-specific. Lets
continue there.

My response below:

>From your question/code below, it is not clear _what_ you aren't seeing as
working alright. Are you observing an error, a use of default partitioner,
etc.? Could you clarify on what exactly isn't working after upgrade
(assuming you've recompiled the code for CDH4 already)?
On Wed, Jan 23, 2013 at 10:28 PM, samir das mohapatra <
[EMAIL PROTECTED]> wrote:

> Hi  All
>    Just we migrated from CDH3 to CDH4 , But after that custom Partitioner
> is note working under CDH4.
>
> Can you tell me why it is like that but in CDH3  same code is working fine.
>
> PARTITONER CODE
> -------------------------------------------
> public static class AgePartitioner extends Partitioner<Text, Text> {
>         static{
>
> System.out.println("&##############################AgePartitioner++++++++++++++++++++++");
>         }
>         @Override
>         public int getPartition(Text key, Text value, int numReduceTasks) {
>
> System.out.println("&##############################AgePartitioner"+value.toString());
>             String [] nameAgeScore = value.toString().split("\t");
>             String age = nameAgeScore[1];
>             int ageInt = Integer.parseInt(age);
>
>             //this is done to avoid performing mod with 0
>             if(numReduceTasks == 0)
>                 return 0;
>
>             //if the age is <20, assign partition 0
>             if(ageInt <=20){
>                 return 0;
>             }
>             //else if the age is between 20 and 50, assign partition 1
>             if(ageInt >20 && ageInt <=50){
>
>                 return 1 % numReduceTasks;
>             }
>             //otherwise assign partition 2
>             else
>                 return 2 % numReduceTasks;
>
>         }
>     }
>
>
>
> DRIVER LEVEL CODE
> -------------------------------------------
>
> job.setPartitionerClass(AgePartitioner.class);
> job.setNumReduceTasks(3);
>
>
>
>
>
--
Harsh J