Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> custom partitioner


Copy link to this message
-
Re: custom partitioner
Typically, numPartitons is used as a modulus in orde to derive a value that
is between zero and strictly less than numPartitons.  That is, key.get() %
numPartitions would yield such a value.

stan
On Mar 25, 2012 11:25 AM, "Harun Raşit ER" <[EMAIL PROTECTED]> wrote:

> public int getPartition(IntWritable key, Chromosome value, int
> numOfPartitions)
>  {
>                int partition = key.get();
>                if (partition < 0 || partition >= numOfPartitions)
>                {
>                        partition = numOfPartitions-1;
>                }
>                System.out.println("partition "+partition );
>                return partition;
> }
>
> I wrote the custom partitioner above. But the problem is about the third
> parameter, numOfPartitions.
>
> It is always "1" in pseudo-distributed mode. I have 4 mappers and 4
> reducers, but only one of the reducers uses the real values. The others
> yield nothing, just empty files.
>
> When I remove the if statement, hadoop complains about the partition number
> as "illegal partition for ...".
>
> How can i set the number of partitions in pseudo-distributed mode?
>
> Thanks.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB