Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Hadoop >> mail # user >> how to resolve conflicts with jar dependencies


Copy link to this message
-
how to resolve conflicts with jar dependencies
hi,

i need to know how to resolve conflicts with jar dependencies.

* first, my job requires Jackson JSON-processor v1.9.11.
* second, the hadoop cluster has Jackson JSON-processor v1.5.2. the
jars are installed in $HADOOP_HOME/lib.

according to this link,
http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/,
there are 3 ways to include 3rd party libraries in a map/reduce (mr)
job.
* use the -libjars flag
* include the dependent libraries in the executing jar file's /lib directory
* put the jars in the $HADOOP_HOME/lib directory

i can report that using -libjars and including the libraries in my
jar's /lib directory "do not work" (in my case of jar conflicts). i
still get a NoSuchMethodException. the only way to get my job to run
is the last option, placing the newer jars in $HADOOP_HOME/lib. the
last option is fine on a sandbox or development instance, but there
are some political difficulties (not only technical) in modifying our
production environment.

my questions/concerns are:
1. how come the -libjars and /lib options do not work? how does class
loading work in mr tasks?
2. is there another option available that i am not aware of to try and
get dependent jars by the job to "overwrite" what's in
$HADOOP_HOME/lib at runtime of the mr tasks?
any help is appreciated. thank you all.
+
Luke Lu 2013-03-12, 23:03
+
Jane Wayne 2013-03-13, 15:19