Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - custom partitioner


Copy link to this message
-
Re: custom partitioner
Stan Rosenberg 2012-03-25, 16:51
Typically, numPartitons is used as a modulus in orde to derive a value that
is between zero and strictly less than numPartitons.  That is, key.get() %
numPartitions would yield such a value.

stan
On Mar 25, 2012 11:25 AM, "Harun Raşit ER" <[EMAIL PROTECTED]> wrote:

> public int getPartition(IntWritable key, Chromosome value, int
> numOfPartitions)
>  {
>                int partition = key.get();
>                if (partition < 0 || partition >= numOfPartitions)
>                {
>                        partition = numOfPartitions-1;
>                }
>                System.out.println("partition "+partition );
>                return partition;
> }
>
> I wrote the custom partitioner above. But the problem is about the third
> parameter, numOfPartitions.
>
> It is always "1" in pseudo-distributed mode. I have 4 mappers and 4
> reducers, but only one of the reducers uses the real values. The others
> yield nothing, just empty files.
>
> When I remove the if statement, hadoop complains about the partition number
> as "illegal partition for ...".
>
> How can i set the number of partitions in pseudo-distributed mode?
>
> Thanks.
>