Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: Get dynamic values in a user defined class from reducer.


Copy link to this message
-
Re: Get dynamic values in a user defined class from reducer.
unmesha sreeveni 2013-12-19, 04:15
Thanks for ur reply Robert Dyer.
Thanks for spending ur valuable time for clearing my doubts.

I need to pass some other values too
Reducer
------------
public  class Reduce extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {

static int cnt =0;
ArrayList<String> ar = new ArrayList<String>();
String data = null;
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
System.out.println("In reducer");
 int sum = 0;
String line = key.toString();
StringTokenizer itr = new StringTokenizer(line);
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));

String data = key+" "+sum;
ar.add(data);
               writeToFile(ar);
ar.add("\n");
int index=Integer.parseInt(itr.nextToken());
String value=itr.nextToken();
String classLabel=itr.nextToken();
int count=sum;

}

public static void writeToFile(ArrayList<String>  text) {
System.out.println("In reduce write to file ");
 try {

cnt++;
C45 id=new C45();
System.out.println("count "+cnt);

Path input = new Path("C45/intermediate"+id.current_index+".txt");
 Configuration conf = new Configuration();
                        FileSystem fs = FileSystem.get(conf);
BufferedWriter bw = new BufferedWriter(
new OutputStreamWriter(fs.create(input, true)));
 System.out.println("Text from Reducer: "+
"C45/intermediate"+id.current_index+".txt"+ text);
System.out.println("file
exists:C45/intermediate"+id.current_index+".txt"+fs.exists(input));

for(String str: text) {
bw.write(str);
}

bw.newLine();
bw.close();
 } catch (Exception e) {
System.out.println("File is not creating in reduce");
}
}

}

Driver
--------
GainRatio  gainObj = new GainRatio();
gainObj.getcount();

GainRatio
--------------

public void getcount()
{
System.out.println("In gainratio --- getcount ");
C45 id=new C45();
FileInputStream fstream;
try {

Configuration config = new Configuration();
FileSystem dfs = FileSystem.get(config);
Path path = new Path("C45/intermediate"+id.current_index+".txt");
System.out.println("file
exists:C45/intermediate"+id.current_index+".txt"+dfs.exists(path));
System.out.println(path);
BufferedReader br=new BufferedReader(new InputStreamReader(dfs.open(path)));
String line;
StringTokenizer itr;
while ((line = br.readLine()) != null)   {
System.out.println("In gain Ratio : "+line);
itr= new StringTokenizer(line);
count[linenumber][0]=itr.nextToken();
count[linenumber][1]=itr.nextToken();
count[linenumber][2]=itr.nextToken();
count[linenumber][3]=itr.nextToken();
int i=linenumber;
linenumber++;
}
System.out.println("outsidec while");
count[linenumber][0]=null;
count[linenumber][1]=null;
count[linenumber][2]=null;
count[linenumber][3]=null;
br.close();
emptyfile(id.current_index);

} catch (Exception e) {

e.printStackTrace();

}
}

Here from reducer many files are created
eg: intermediate0.txt
     .....upto intermediate7.txt according to my input dataset.

But here only intermediate0.txt is created all other creation of file get
failed.
Is that the same reason you explained before.

For this also should i go for couters?

On Wed, Dec 18, 2013 at 11:09 PM, Robert Dyer <[EMAIL PROTECTED]> wrote:

> Generally speaking, static fields are not useful in Hadoop.
>
> The issue you are seeing is that the reducer is running in a separate VM
> (possibly on a different node!) and thus the static value you are reading
> inside of Mid is actually a separate instantiation of that class and field.
>
> If you have an integer you need to set and read from the driver/mid, you
> can use a counter.  E.g.:
>
>
> http://www.philippeadjiman.com/blog/2010/01/07/hadoop-tutorial-series-issue-3-counters-in-action/
>
> - Robert
>
> On Wed, Dec 18, 2013 at 4:02 AM, unmesha sreeveni <[EMAIL PROTECTED]>wrote:
>
>> Can any one pls suggest a good way
>> My scenario:
>> i hav
>> 1.Driver class
>> 2.Mapper class
>> 3.reducer class
>> 4.Mid class
>> After completing mapper it goes to reduce. From reducer it will be going
*Thanks & Regards*

Unmesha Sreeveni U.B

*Junior Developer*