Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Re: mapper combiner and partitioner for particular dataset


Copy link to this message
-
Re: mapper combiner and partitioner for particular dataset
The MultipleInputs class only supports mapper configuration per dataset. It
does not let you specify a partitioner and combiner as well. You will need
a custom written "high level" partitioner and combiner that can create
multiple instances of sub-partitioners/combiners and use the most likely
one based on their input's characteristics (such as instance type, some
tag, config., etc.).
On Sun, Mar 3, 2013 at 4:07 PM, Vikas Jadhav <[EMAIL PROTECTED]>wrote:

>
>
>
>
> Hello
>
> 1)  I have multiple types of datasets as input to my hadoop job
>
> i want write my own inputformat (Exa. MyTableInputformat)
>   and how to specify mapper partitioner combiner per dataset manner
>  I know MultiFileInputFormat class but if i want to asscoite combiner and
> partitioner class
> it wont help. it only sets mapper class for per dataset manner.
>
> 2)  Also i am looking MapTask.java file from source code
>
> just want to know where does mapper partitioner and combiner classes are
> set for particular filesplit
> while executing job
>
> Thank You
>
> --
> *
> *
> *
>
>  Thanx and Regards*
> * Vikas Jadhav*
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB