Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Re: reducer gets values with empty attributes


Copy link to this message
-
Re: reducer gets values with empty attributes
alxsss@... 2013-05-01, 18:12
Hi,

Here is the map and reduce part of the code

  public void map(Text key, Writable  value, OutputCollector<Text, Document> output, Reporter reporter) throws IOException {

      ParseData parseData = (ParseData) value;
      Metadata parseMeta = parseData.getParseMeta();
      if(parseMeta.getValues("id").length>0)
      {
        
        String[] id = parseMeta.getValues("id");      
        String [] ids=id[0].split(",");      
        int len=ids.length;      
        for(int i=0;i<len;i++)
        {
           Document doc = new Document();
           doc.add("id", ids[i]);
           output.collect(new Text(ids[i]), doc);
          //    for testing
            List list=doc.getField("id").getValues();
           for(int j=0;j<list.size();j++){
              System.out.println("map:sending to reducer id field for key "+ids[i]+" as "+list.get(j));
            }
           //
         }
     }
  }

 

 
public void reduce(Text key, Iterator<Document> values, OutputCollector<Text, Document> output, Reporter reporter) throws IOException{
    Document doc=values.next();
 
    System.out.println("reducer keys are  "+key +" and size of doc is  "+doc.getField("id").getValues().size() +"\n");

    List list=doc.getField("id").getValues();
    System.out.println("reducer: list size is "+list.size() +"\n");
    for(int j=0;j<list.size();j++){
    System.out.println("reduce:id field for key "+key+" is "+list.get(j));
    }
    //output.collect(key, doc);
  }
 System.out.println in map function outputs expected values for id field for each document. However, System.out.println in reducer function outputs keys but no values for id field.
Thanks in advance.
Alex.

-----Original Message-----
From: Mahesh Balija <[EMAIL PROTECTED]>
To: user <[EMAIL PROTECTED]>
Sent: Mon, Apr 29, 2013 11:33 pm
Subject: Re: reducer gets values with empty attributes

Hi Alex,
             Can you please attach your code? and the sample input data.

Best,
Mahesh Balija,

Calsoft Labs.
On Tue, Apr 30, 2013 at 2:29 AM,  <[EMAIL PROTECTED]> wrote:

Hello,

I try to write mapreduce program in hadoop -1.0.4. using mapred libs. I have a map function which gets
keys and creates a different object with a few attributes like id and etc and passes it to reducer function using

 output.collect(key, value);

Reducer gets keys, but values  has empty  fields (like id and etc) , although map correctly assigns these fields to each value.

Any ideas why this happens?

Thanks.
Alex.