Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> Lib conflicts


Copy link to this message
-
Lib conflicts
Hi all,

I have a jar that uses the Hadoop API to launch various remote mapreduce
jobs (ie, im not using the command-line to initiate the job). The service
jar that executes the various jobs is built with maven's
"jar-with-dependencies".

My jobs all run fine except one that uses commons-codec 1.7, I get:

FATAL org.apache.hadoop.mapred.Child: Error running child :
java.lang.NoSuchMethodError:
org.apache.commons.codec.binary.Base64.encodeAsString([B)Ljava/lang/String;

I think this is because my jar is including commons-codec 1.7 whereas my
Hadoop install's lib has commons-codec 1.4 ...

Is their any way to instruct Hadoop to use the distributed commons-codec
1.7 (I assume this is distributed as a job dependency) rather than the
commons-codec 1.4 in the hadoop 1.0.3 core lib?

Removing commons-codec-1.4.jar from my Hadoop library folder did seem to
solve the problem for a bit, but is not working on another VM. Replacing
the 1.4 jar with the 1.7 does seem to fix the problem but this doesn't seem
too sane. Hopefully there is a better alternative.

Thanks!
+
Harsh J 2012-10-03, 11:39
+
Ben Rycroft 2012-10-03, 11:45
+
Jay Vyas 2012-10-03, 12:01
+
Michael Segel 2012-10-03, 11:32
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB