Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Lib conflicts


Copy link to this message
-
Re: Lib conflicts
Ben Rycroft 2012-10-03, 11:45
Hi guys,

Thanks the speedy replies.

I will try Harsh's suggestion and see if this solves the issue, if not I
will just do what Michael suggested and replace the jars on each of the
nodes.

Thanks again!

On Wed, Oct 3, 2012 at 12:39 PM, Harsh J <[EMAIL PROTECTED]> wrote:

> Hi Ben,
>
> As long as the switch of libraries doesn't impact the execution of the
> Child task code itself, for Apache Hadoop 1.x, using the config
> "mapreduce.user.classpath.first" set to true may solve your trouble.
>
> On Wed, Oct 3, 2012 at 4:51 PM, Ben Rycroft <[EMAIL PROTECTED]> wrote:
> > Hi all,
> >
> > I have a jar that uses the Hadoop API to launch various remote mapreduce
> > jobs (ie, im not using the command-line to initiate the job). The service
> > jar that executes the various jobs is built with maven's
> > "jar-with-dependencies".
> >
> > My jobs all run fine except one that uses commons-codec 1.7, I get:
> >
> > FATAL org.apache.hadoop.mapred.Child: Error running child :
> > java.lang.NoSuchMethodError:
> >
> org.apache.commons.codec.binary.Base64.encodeAsString([B)Ljava/lang/String;
> >
> > I think this is because my jar is including commons-codec 1.7 whereas my
> > Hadoop install's lib has commons-codec 1.4 ...
> >
> > Is their any way to instruct Hadoop to use the distributed commons-codec
> 1.7
> > (I assume this is distributed as a job dependency) rather than the
> > commons-codec 1.4 in the hadoop 1.0.3 core lib?
> >
> > Removing commons-codec-1.4.jar from my Hadoop library folder did seem to
> > solve the problem for a bit, but is not working on another VM. Replacing
> the
> > 1.4 jar with the 1.7 does seem to fix the problem but this doesn't seem
> too
> > sane. Hopefully there is a better alternative.
> >
> > Thanks!
>
>
>
> --
> Harsh J
>