Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Partitioner in Hadoop 0.20


Copy link to this message
-
Re: Partitioner in Hadoop 0.20
On 08/04/2010 01:55 PM, Wilkes, Chris wrote:
> On Aug 4, 2010, at 10:50 AM, David Rosenstrauch wrote:
>
>> On 08/04/2010 12:30 PM, Owen O'Malley wrote:
>>>
>>> On Aug 4, 2010, at 8:38 AM, David Rosenstrauch wrote:
>>>
>>>> Anyone know if there's any particular reason why the new Partitioner
>>>> class doesn't implement JobConfigurable? (And, if not, whether there's
>>>> any plans to fix this omission?) We're working on a somewhat complex
>>>> partitioner, and it would be extremely helpful to be able to pass it
>>>> some parms via the jobconf.
>>>
>>> The short answer is that it doesn't need to. If you make your
>>> partitioner either Configured or JobConfigurable, it will be configured.
>>> The API class doesn't depend on it precisely because it is not required
>>> for all partitioners.
>>>
>>> -- Owen
>>
>> ? Not sure I understand correctly ... can you pls clarify?
>>
>> So if I make my custom partitioner implement JobConfigurable, then its
>> configure(JobConf) method will automagically get called and allow me
>> to configure it with info in the jobConf that's passed in? (Note that
>> making it extend from Configured is not an option, since it needs to
>> extend from org.apache.hadoop.mapreduce.Partitioner.)
>>
>
>
> The partitioner is instantiated by ReflectionUtils.newInstance(clazz,
> job) , that calls the setConfiguration() on the newly created object if
> it implements Configurable
>
> Chris

So my partitioner needs to implement Configurable, then not
JobConfigurable.  Tnx much!

DR