-Re: custom partitioner
Stan Rosenberg 2012-03-25, 16:51
Typically, numPartitons is used as a modulus in orde to derive a value that
is between zero and strictly less than numPartitons. That is, key.get() %
numPartitions would yield such a value.
On Mar 25, 2012 11:25 AM, "Harun Raşit ER" <[EMAIL PROTECTED]> wrote:
> public int getPartition(IntWritable key, Chromosome value, int
> int partition = key.get();
> if (partition < 0 || partition >= numOfPartitions)
> partition = numOfPartitions-1;
> System.out.println("partition "+partition );
> return partition;
> I wrote the custom partitioner above. But the problem is about the third
> parameter, numOfPartitions.
> It is always "1" in pseudo-distributed mode. I have 4 mappers and 4
> reducers, but only one of the reducers uses the real values. The others
> yield nothing, just empty files.
> When I remove the if statement, hadoop complains about the partition number
> as "illegal partition for ...".
> How can i set the number of partitions in pseudo-distributed mode?