-Re: Partitioner in Hadoop 0.20
David Rosenstrauch 2010-08-04, 17:58
On 08/04/2010 01:55 PM, Wilkes, Chris wrote:
> On Aug 4, 2010, at 10:50 AM, David Rosenstrauch wrote:
>> On 08/04/2010 12:30 PM, Owen O'Malley wrote:
>>> On Aug 4, 2010, at 8:38 AM, David Rosenstrauch wrote:
>>>> Anyone know if there's any particular reason why the new Partitioner
>>>> class doesn't implement JobConfigurable? (And, if not, whether there's
>>>> any plans to fix this omission?) We're working on a somewhat complex
>>>> partitioner, and it would be extremely helpful to be able to pass it
>>>> some parms via the jobconf.
>>> The short answer is that it doesn't need to. If you make your
>>> partitioner either Configured or JobConfigurable, it will be configured.
>>> The API class doesn't depend on it precisely because it is not required
>>> for all partitioners.
>>> -- Owen
>> ? Not sure I understand correctly ... can you pls clarify?
>> So if I make my custom partitioner implement JobConfigurable, then its
>> configure(JobConf) method will automagically get called and allow me
>> to configure it with info in the jobConf that's passed in? (Note that
>> making it extend from Configured is not an option, since it needs to
>> extend from org.apache.hadoop.mapreduce.Partitioner.)
> The partitioner is instantiated by ReflectionUtils.newInstance(clazz,
> job) , that calls the setConfiguration() on the newly created object if
> it implements Configurable
So my partitioner needs to implement Configurable, then not
JobConfigurable. Tnx much!