Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> how to resolve conflicts with jar dependencies


Copy link to this message
-
Re: how to resolve conflicts with jar dependencies
The problem is resolved in the next release of hadoop (2.0.3-alpha cf.
MAPREDUCE-1700)

For hadoop 1.x based releases/distributions, put
-Dmapreduce.user.classpath.first=true on the hadoop command line and/or
client config
On Tue, Mar 12, 2013 at 6:49 AM, Jane Wayne <[EMAIL PROTECTED]>wrote:

> hi,
>
> i need to know how to resolve conflicts with jar dependencies.
>
> * first, my job requires Jackson JSON-processor v1.9.11.
> * second, the hadoop cluster has Jackson JSON-processor v1.5.2. the
> jars are installed in $HADOOP_HOME/lib.
>
> according to this link,
>
> http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/
> ,
> there are 3 ways to include 3rd party libraries in a map/reduce (mr)
> job.
> * use the -libjars flag
> * include the dependent libraries in the executing jar file's /lib
> directory
> * put the jars in the $HADOOP_HOME/lib directory
>
> i can report that using -libjars and including the libraries in my
> jar's /lib directory "do not work" (in my case of jar conflicts). i
> still get a NoSuchMethodException. the only way to get my job to run
> is the last option, placing the newer jars in $HADOOP_HOME/lib. the
> last option is fine on a sandbox or development instance, but there
> are some political difficulties (not only technical) in modifying our
> production environment.
>
> my questions/concerns are:
> 1. how come the -libjars and /lib options do not work? how does class
> loading work in mr tasks?
> 2. is there another option available that i am not aware of to try and
> get dependent jars by the job to "overwrite" what's in
> $HADOOP_HOME/lib at runtime of the mr tasks?
>
>
> any help is appreciated. thank you all.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB