Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: Custom Partitioner is now working in CDH4


Copy link to this message
-
Re: Custom Partitioner is now working in CDH4
Moving to cdh-user@ as you seem to indicate this is CDH-specific. Lets
continue there.

My response below:

>From your question/code below, it is not clear _what_ you aren't seeing as
working alright. Are you observing an error, a use of default partitioner,
etc.? Could you clarify on what exactly isn't working after upgrade
(assuming you've recompiled the code for CDH4 already)?
On Wed, Jan 23, 2013 at 10:28 PM, samir das mohapatra <
[EMAIL PROTECTED]> wrote:

> Hi  All
>    Just we migrated from CDH3 to CDH4 , But after that custom Partitioner
> is note working under CDH4.
>
> Can you tell me why it is like that but in CDH3  same code is working fine.
>
> PARTITONER CODE
> -------------------------------------------
> public static class AgePartitioner extends Partitioner<Text, Text> {
>         static{
>
> System.out.println("&##############################AgePartitioner++++++++++++++++++++++");
>         }
>         @Override
>         public int getPartition(Text key, Text value, int numReduceTasks) {
>
> System.out.println("&##############################AgePartitioner"+value.toString());
>             String [] nameAgeScore = value.toString().split("\t");
>             String age = nameAgeScore[1];
>             int ageInt = Integer.parseInt(age);
>
>             //this is done to avoid performing mod with 0
>             if(numReduceTasks == 0)
>                 return 0;
>
>             //if the age is <20, assign partition 0
>             if(ageInt <=20){
>                 return 0;
>             }
>             //else if the age is between 20 and 50, assign partition 1
>             if(ageInt >20 && ageInt <=50){
>
>                 return 1 % numReduceTasks;
>             }
>             //otherwise assign partition 2
>             else
>                 return 2 % numReduceTasks;
>
>         }
>     }
>
>
>
> DRIVER LEVEL CODE
> -------------------------------------------
>
> job.setPartitionerClass(AgePartitioner.class);
> job.setNumReduceTasks(3);
>
>
>
>
>
--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB