Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS, mail # user - Re: mapper combiner and partitioner for particular dataset


Copy link to this message
-
Re: mapper combiner and partitioner for particular dataset
Harsh J 2013-03-03, 11:28
The MultipleInputs class only supports mapper configuration per dataset. It
does not let you specify a partitioner and combiner as well. You will need
a custom written "high level" partitioner and combiner that can create
multiple instances of sub-partitioners/combiners and use the most likely
one based on their input's characteristics (such as instance type, some
tag, config., etc.).
On Sun, Mar 3, 2013 at 4:07 PM, Vikas Jadhav <[EMAIL PROTECTED]>wrote:

>
>
>
>
> Hello
>
> 1)  I have multiple types of datasets as input to my hadoop job
>
> i want write my own inputformat (Exa. MyTableInputformat)
>   and how to specify mapper partitioner combiner per dataset manner
>  I know MultiFileInputFormat class but if i want to asscoite combiner and
> partitioner class
> it wont help. it only sets mapper class for per dataset manner.
>
> 2)  Also i am looking MapTask.java file from source code
>
> just want to know where does mapper partitioner and combiner classes are
> set for particular filesplit
> while executing job
>
> Thank You
>
> --
> *
> *
> *
>
>  Thanx and Regards*
> * Vikas Jadhav*
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>

--
Harsh J