Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Partitioners - How to know if they are working


Copy link to this message
-
Re: Partitioners - How to know if they are working
On 02/16/2012 12:49 PM, [EMAIL PROTECTED] wrote:
> Hello All,
> I wrote my own partitioner and I would like to see if it's working.
> By printing the return of method getPartition I could see that the partitions were different, but were they really working? To answer that I got the keys that every reducer task processed and that was what I expected. It seems my partitioner is working properly. But not easy to discover though.
> Does anyone know if there is an easier way to see if your customized partitioner is working? For instance, a counter that shows how many partitioners a map generated or a reducer received?
> Thanks in advance,
> Fabio Almeida

At my last job we wrote a custom partitioner, and we tested it out
completely outside of Hadoop using standard JUnit unit tests.

HTH,

DR
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB