Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> setOutputValueClass(*.class) method


Copy link to this message
-
RE: setOutputValueClass(*.class) method
Bertand

I am reading my input by declaring it in the driver as follows:

conf.setInputFormat(SequenceFileInputFormat.class);
In the mapper:
public void map(LongWritable key, SomeInterfaceClass value,
                        OutputCollector<Text, SomeInterfaceClass > output, Reporter reporter)
                                throws IOException {
                SomeInterfaceClass o = null;
                        if ( value instanceof classA) {
                          o = (classA)value;
                  }
                        if ( value instanceof classB) {
                          o = (classB)value;
                  }
                output.collect(new Text(o.get_property_id().toString()), o);
        } // end of map()
     } // end of static class

As u can see "o" can be an instance of classA or classB, where both implement SomeInerfaceClass.

However, when I try to write in the driver:
Conf.setOutputValueClass(SomeIterfaceClass.class);
And run the job I get the following error

Error: java.io.IOException: Type mismatch in value from map: expected SomeInterfaceClass, received ClassA

I wished I could change the outputClass name depending on the instance of "o" to either ClassA or ClassB.

From: Bertrand Dechoux [mailto:[EMAIL PROTECTED]]
Sent: Thursday, October 11, 2012 2:12 PM
To: [EMAIL PROTECTED]
Subject: Re: setOutputValueClass(*.class) method

Hi Andy,

It should be called within the setup of your job and it should NOT be called within the Mapper.
You could compile the call within the mapper of course. At best it should do nothing. I would say in practice the job will fail. But that's something that you have to verify if you are curious.

Back to your original problem, the mapper need to output the same kind of object in order to be able to unserialize them for the mapper. (More generally the serialization strategy must be able to figure out what was serialized.) It does not mean your problem is not solvable but it means that you will need to come up with a wrapper or a common structure.

How do you read different kind of objects with the Mapper? Maybe you could use the same approach.

Regards

Bertrand
On Thu, Oct 11, 2012 at 7:49 PM, Kartashov, Andy <[EMAIL PROTECTED]<mailto:[EMAIL PROTECTED]>> wrote:
Guys,

Is there another way to set the output from within the Mapper?

My Mapper reads from various serialised files and generates different type of objects for values depending on the returned value of instanceof. I wanted to change the class name within the Mapper as opposed to the driver class. Anyway of doing it?

Thanks,
Andy

NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel

--
Bertrand Dechoux
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB