|
|
-
Re: Custom Partitioner is now working in CDH4Harsh J 2013-01-23, 17:03
Moving to cdh-user@ as you seem to indicate this is CDH-specific. Lets
continue there. My response below: >From your question/code below, it is not clear _what_ you aren't seeing as working alright. Are you observing an error, a use of default partitioner, etc.? Could you clarify on what exactly isn't working after upgrade (assuming you've recompiled the code for CDH4 already)? On Wed, Jan 23, 2013 at 10:28 PM, samir das mohapatra < [EMAIL PROTECTED]> wrote: > Hi All > Just we migrated from CDH3 to CDH4 , But after that custom Partitioner > is note working under CDH4. > > Can you tell me why it is like that but in CDH3 same code is working fine. > > PARTITONER CODE > ------------------------------------------- > public static class AgePartitioner extends Partitioner<Text, Text> { > static{ > > System.out.println("#############################AgePartitioner++++++++++++++++++++++"); > } > @Override > public int getPartition(Text key, Text value, int numReduceTasks) { > > System.out.println("#############################AgePartitioner"+value.toString()); > String [] nameAgeScore = value.toString().split("\t"); > String age = nameAgeScore[1]; > int ageInt = Integer.parseInt(age); > > //this is done to avoid performing mod with 0 > if(numReduceTasks == 0) > return 0; > > //if the age is <20, assign partition 0 > if(ageInt <=20){ > return 0; > } > //else if the age is between 20 and 50, assign partition 1 > if(ageInt >20 && ageInt <=50){ > > return 1 % numReduceTasks; > } > //otherwise assign partition 2 > else > return 2 % numReduceTasks; > > } > } > > > > DRIVER LEVEL CODE > ------------------------------------------- > > job.setPartitionerClass(AgePartitioner.class); > job.setNumReduceTasks(3); > > > > > -- Harsh J |