Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: passing arguments to hadoop job


Copy link to this message
-
Re: passing arguments to hadoop job
Satbeer Lamba 2013-01-22, 01:55
Please be sure that you are getting the value of baseSum in reducer by
adding a logger or syso.

Also consider removing static in declaration of baseSum as it would add
counts of previous keys.
On Jan 22, 2013 7:17 AM, "jamal sasha" <[EMAIL PROTECTED]> wrote:

> The second one.
> If the word hello appears once, its count is  2000001.
> :)
>
>
> On Mon, Jan 21, 2013 at 5:40 PM, Satbeer Lamba <[EMAIL PROTECTED]>wrote:
>
>> Do you mean to say you want to count the words from 200000 th line
>> onwards?
>>
>> OR
>>
>> You want to start counting from 2000000?
>> For example if HELLO appears once it's count is 2000001.
>>
>> Please clarify
>> On Jan 22, 2013 5:22 AM, "jamal sasha" <[EMAIL PROTECTED]> wrote:
>>
>>> Hi,
>>>   Lets say I have the standard helloworld program
>>>
>>> http://hadoop.apache.org/docs/r0.17.0/mapred_tutorial.html#Example%3A+WordCount+v2.0
>>>
>>> Now, lets say, I want to start the counting not from zero but from
>>> 200000.
>>> So my reference line is 200000.
>>>
>>> I modified the Reduce code as following:
>>>  public static class Reduce extends MapReduceBase implements
>>> Reducer<Text, IntWritable, Text, IntWritable> {
>>>      *private static int baseSum ;*
>>> *      public void configure(JobConf job){*
>>> *      baseSum = Integer.parseInt(job.get("basecount"));*
>>> *      *
>>> *      }*
>>>        public void reduce(Text key, Iterator<IntWritable> values,
>>> OutputCollector<Text, IntWritable> output, Reporter reporter) throws
>>> IOException {
>>>          int sum =* baseSum*;
>>>         while (values.hasNext()) {
>>>           sum += values.next().get();
>>>          }
>>>         output.collect(key, new IntWritable(sum));
>>>       }
>>>      }
>>>
>>>
>>> And in main added:
>>>    conf.setInt("basecount",200000);
>>>
>>>
>>>
>>> So my hope was this should have done the trick..
>>> But its not working. the code is running normally :(
>>> How do i resolve this?
>>> Thanks
>>>
>>
>