Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce, mail # user - Lib conflicts


Copy link to this message
-
Lib conflicts
Ben Rycroft 2012-10-03, 11:21
Hi all,

I have a jar that uses the Hadoop API to launch various remote mapreduce
jobs (ie, im not using the command-line to initiate the job). The service
jar that executes the various jobs is built with maven's
"jar-with-dependencies".

My jobs all run fine except one that uses commons-codec 1.7, I get:

FATAL org.apache.hadoop.mapred.Child: Error running child :
java.lang.NoSuchMethodError:
org.apache.commons.codec.binary.Base64.encodeAsString([B)Ljava/lang/String;

I think this is because my jar is including commons-codec 1.7 whereas my
Hadoop install's lib has commons-codec 1.4 ...

Is their any way to instruct Hadoop to use the distributed commons-codec
1.7 (I assume this is distributed as a job dependency) rather than the
commons-codec 1.4 in the hadoop 1.0.3 core lib?

Removing commons-codec-1.4.jar from my Hadoop library folder did seem to
solve the problem for a bit, but is not working on another VM. Replacing
the 1.4 jar with the 1.7 does seem to fix the problem but this doesn't seem
too sane. Hopefully there is a better alternative.

Thanks!
+
Harsh J 2012-10-03, 11:39
+
Ben Rycroft 2012-10-03, 11:45
+
Jay Vyas 2012-10-03, 12:01
+
Michael Segel 2012-10-03, 11:32