Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> Task does not enter reduce function after secondary sort


Copy link to this message
-
Re: Task does not enter reduce function after secondary sort
Only to be clear but the @Override annotation has no impact by itself.
However if you put it then the compiler can check for you that you are
indeed overriding a method. If you don't use the annotation, you might be
defining another function (with another signature) which wouldn't be
called. And so the identity (the default implementation) would be used
instead.

Regards

Bertrand

On Mon, Nov 5, 2012 at 11:27 AM, Aseem Anand <[EMAIL PROTECTED]> wrote:

> Hey,
> That change coupled with a few minor issues seemed to work. Though its
> strange that mapreduce programs I wrote with the same API were working till
> now without Override. Thanks :).
>
> Thanks,
> Aseem
> On Mon, Nov 5, 2012 at
>  3:33 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>
>> Yep - it will show an error since your reduce(…) signature is wrong
>> for the new API:
>>
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Reducer.html#reduce(KEYIN,%20java.lang.Iterable,%20org.apache.hadoop.mapreduce.Reducer.Context)
>>
>> Chuck the Reporter object (its an Old API thing, now built into
>> Context itself) and transform it into:
>>
>> @Override
>> public void reduce(Text key, Iterable<NullWritable> values, Context
>> output) {
>>     …
>> }
>>
>> … and your IDE shouldn't complain anymore.
>>
>> On Mon, Nov 5, 2012 at 2:45 AM, Aseem Anand <[EMAIL PROTECTED]>
>> wrote:
>> > Hey,
>> > Here are code snippets.
>> >
>> > In the driver class :
>> >         job.setMapperClass(SkyzKnnMapperT.class);
>> >         job.setReducerClass(SkyzKnnReducer.class);
>> >         job.setGroupingComparatorClass(GroupComparator.class);
>> >         job.setPartitionerClass(MyPartitioner.class);
>> >         job.setSortComparatorClass(KeyComparator.class);
>> >         job.setMapOutputKeyClass(Text.class);
>> >         job.setMapOutputValueClass(NullWritable.class);
>> >         job.setOutputKeyClass(Text.class);
>> >         job.setOutputValueClass(Text.class);
>> >
>> > public class GroupComparator extends WritableComparator {
>> >
>> > protected GroupComparator() {
>> > super(Text.class, true);
>> > }
>> >
>> > @Override
>> > public int compare(WritableComparable w1, WritableComparable w2) {
>> >
>> > //consider only zone and day part of the key
>> > Text t1 = (Text) w1;
>> > Text t2 = (Text) w2;
>> > String[] t1Items = t1.toString().split(":");
>> > String[] t2Items = t2.toString().split(":");
>> > int comp = t1Items[0].compareTo(t2Items[0]);
>> > System.out.println("GROUP" + comp);
>> > return comp;
>> >
>> > }
>> > }
>> > public class SkyzKnnReducer extends Reducer<Text,Iterable,Text,Text> {
>> > public void reduce(Text key, Iterable<NullWritable> values,
>> > Context output, Reporter reporter)
>> > throws IOException, InterruptedException {
>> >             String t = key.toString();
>> >             t = "HELLO" + t;
>> >             output.write(new Text(t),new Text(t));
>> >
>> > }
>> > }
>> >
>> > The composite key is of the form A:Rest_of_text where A is the natural
>> key.
>> >
>> > Override annotation to this reduce method shows an error in Eclipse.
>> What
>> > else could be going wrong ?
>> >
>> > Thanks,
>> > Aseem
>> > On Mon, Nov 5, 2012 at 2:33 AM, Harsh J <[EMAIL PROTECTED]> wrote:
>> >>
>> >> Sounds like an override issue to me. If you can share your code, we
>> >> can take a quick look - otherwise, try annotating your reduce(…)
>> >> method with @Override and recompiling to see if it really is the right
>> >> signature Java expects.
>> >>
>> >> On Mon, Nov 5, 2012 at 1:48 AM, Aseem Anand <[EMAIL PROTECTED]>
>> wrote:
>> >> > Hi,
>> >> > I am using a Secondary Sort for my Hadoop program. My map function
>> emits
>> >> > (Text,NullWritable) where Text contains the composite key and
>> >> > appropriate
>> >> > comparison functions are made and a custom Partitioner . These seem
>> to
>> >> > be
>> >> > working fine.
>> >> >
>> >> > I have been struggling with the problem that these values are not
>> being
>> >> > received by the reduce function and instead automatically get
Bertrand Dechoux
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB