Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Re: reducer gets values with empty attributes


+
Mahesh Balija 2013-04-30, 06:33
Copy link to this message
-
Re: reducer gets values with empty attributes
Hi,

Here is the map and reduce part of the code

  public void map(Text key, Writable  value, OutputCollector<Text, Document> output, Reporter reporter) throws IOException {

      ParseData parseData = (ParseData) value;
      Metadata parseMeta = parseData.getParseMeta();
      if(parseMeta.getValues("id").length>0)
      {
        
        String[] id = parseMeta.getValues("id");      
        String [] ids=id[0].split(",");      
        int len=ids.length;      
        for(int i=0;i<len;i++)
        {
           Document doc = new Document();
           doc.add("id", ids[i]);
           output.collect(new Text(ids[i]), doc);
          //    for testing
            List list=doc.getField("id").getValues();
           for(int j=0;j<list.size();j++){
              System.out.println("map:sending to reducer id field for key "+ids[i]+" as "+list.get(j));
            }
           //
         }
     }
  }

 

 
public void reduce(Text key, Iterator<Document> values, OutputCollector<Text, Document> output, Reporter reporter) throws IOException{
    Document doc=values.next();
 
    System.out.println("reducer keys are  "+key +" and size of doc is  "+doc.getField("id").getValues().size() +"\n");

    List list=doc.getField("id").getValues();
    System.out.println("reducer: list size is "+list.size() +"\n");
    for(int j=0;j<list.size();j++){
    System.out.println("reduce:id field for key "+key+" is "+list.get(j));
    }
    //output.collect(key, doc);
  }
 System.out.println in map function outputs expected values for id field for each document. However, System.out.println in reducer function outputs keys but no values for id field.
Thanks in advance.
Alex.

-----Original Message-----
From: Mahesh Balija <[EMAIL PROTECTED]>
To: user <[EMAIL PROTECTED]>
Sent: Mon, Apr 29, 2013 11:33 pm
Subject: Re: reducer gets values with empty attributes

Hi Alex,
             Can you please attach your code? and the sample input data.

Best,
Mahesh Balija,

Calsoft Labs.
On Tue, Apr 30, 2013 at 2:29 AM,  <[EMAIL PROTECTED]> wrote:

Hello,

I try to write mapreduce program in hadoop -1.0.4. using mapred libs. I have a map function which gets
keys and creates a different object with a few attributes like id and etc and passes it to reducer function using

 output.collect(key, value);

Reducer gets keys, but values  has empty  fields (like id and etc) , although map correctly assigns these fields to each value.

Any ideas why this happens?

Thanks.
Alex.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB