Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Partitioners - How to know if they are working

Copy link to this message
Re: Partitioners - How to know if they are working
On 02/16/2012 12:49 PM, [EMAIL PROTECTED] wrote:
> Hello All,
> I wrote my own partitioner and I would like to see if it's working.
> By printing the return of method getPartition I could see that the partitions were different, but were they really working? To answer that I got the keys that every reducer task processed and that was what I expected. It seems my partitioner is working properly. But not easy to discover though.
> Does anyone know if there is an easier way to see if your customized partitioner is working? For instance, a counter that shows how many partitioners a map generated or a reducer received?
> Thanks in advance,
> Fabio Almeida

At my last job we wrote a custom partitioner, and we tested it out
completely outside of Hadoop using standard JUnit unit tests.