Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Re: Partitioner in Hadoop 0.20


Copy link to this message
-
Re: Partitioner in Hadoop 0.20
On 08/04/2010 01:55 PM, Wilkes, Chris wrote:
> On Aug 4, 2010, at 10:50 AM, David Rosenstrauch wrote:
>
>> On 08/04/2010 12:30 PM, Owen O'Malley wrote:
>>>
>>> On Aug 4, 2010, at 8:38 AM, David Rosenstrauch wrote:
>>>
>>>> Anyone know if there's any particular reason why the new Partitioner
>>>> class doesn't implement JobConfigurable? (And, if not, whether there's
>>>> any plans to fix this omission?) We're working on a somewhat complex
>>>> partitioner, and it would be extremely helpful to be able to pass it
>>>> some parms via the jobconf.
>>>
>>> The short answer is that it doesn't need to. If you make your
>>> partitioner either Configured or JobConfigurable, it will be configured.
>>> The API class doesn't depend on it precisely because it is not required
>>> for all partitioners.
>>>
>>> -- Owen
>>
>> ? Not sure I understand correctly ... can you pls clarify?
>>
>> So if I make my custom partitioner implement JobConfigurable, then its
>> configure(JobConf) method will automagically get called and allow me
>> to configure it with info in the jobConf that's passed in? (Note that
>> making it extend from Configured is not an option, since it needs to
>> extend from org.apache.hadoop.mapreduce.Partitioner.)
>>
>
>
> The partitioner is instantiated by ReflectionUtils.newInstance(clazz,
> job) , that calls the setConfiguration() on the newly created object if
> it implements Configurable
>
> Chris

So my partitioner needs to implement Configurable, then not
JobConfigurable.  Tnx much!

DR
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB