Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Re: passing arguments to hadoop job


+
Satbeer Lamba 2013-01-22, 01:55
Copy link to this message
-
Re: passing arguments to hadoop job
jamal sasha 2013-01-22, 02:04
Hi,
  Thanks for taking out time.
Will this not work : conf.setInt("basecount",200000); ??
I am not sure how to add loger or syso (new to both java and hadoop :( )

On Mon, Jan 21, 2013 at 5:55 PM, Satbeer Lamba <[EMAIL PROTECTED]>wrote:

> Please be sure that you are getting the value of baseSum in reducer by
> adding a logger or syso.
>
> Also consider removing static in declaration of baseSum as it would add
> counts of previous keys.
> On Jan 22, 2013 7:17 AM, "jamal sasha" <[EMAIL PROTECTED]> wrote:
>
>> The second one.
>> If the word hello appears once, its count is  2000001.
>> :)
>>
>>
>> On Mon, Jan 21, 2013 at 5:40 PM, Satbeer Lamba <[EMAIL PROTECTED]>wrote:
>>
>>> Do you mean to say you want to count the words from 200000 th line
>>> onwards?
>>>
>>> OR
>>>
>>> You want to start counting from 2000000?
>>> For example if HELLO appears once it's count is 2000001.
>>>
>>> Please clarify
>>> On Jan 22, 2013 5:22 AM, "jamal sasha" <[EMAIL PROTECTED]> wrote:
>>>
>>>> Hi,
>>>>   Lets say I have the standard helloworld program
>>>>
>>>> http://hadoop.apache.org/docs/r0.17.0/mapred_tutorial.html#Example%3A+WordCount+v2.0
>>>>
>>>> Now, lets say, I want to start the counting not from zero but from
>>>> 200000.
>>>> So my reference line is 200000.
>>>>
>>>> I modified the Reduce code as following:
>>>>  public static class Reduce extends MapReduceBase implements
>>>> Reducer<Text, IntWritable, Text, IntWritable> {
>>>>      *private static int baseSum ;*
>>>> *      public void configure(JobConf job){*
>>>> *      baseSum = Integer.parseInt(job.get("basecount"));*
>>>> *      *
>>>> *      }*
>>>>        public void reduce(Text key, Iterator<IntWritable> values,
>>>> OutputCollector<Text, IntWritable> output, Reporter reporter) throws
>>>> IOException {
>>>>          int sum =* baseSum*;
>>>>         while (values.hasNext()) {
>>>>           sum += values.next().get();
>>>>          }
>>>>         output.collect(key, new IntWritable(sum));
>>>>       }
>>>>      }
>>>>
>>>>
>>>> And in main added:
>>>>    conf.setInt("basecount",200000);
>>>>
>>>>
>>>>
>>>> So my hope was this should have done the trick..
>>>> But its not working. the code is running normally :(
>>>> How do i resolve this?
>>>> Thanks
>>>>
>>>
>>
+
Satbeer Lamba 2013-01-22, 02:27
+
Hemanth Yamijala 2013-01-22, 04:29
+
jamal sasha 2013-01-22, 05:49