Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> how to resolve conflicts with jar dependencies

Copy link to this message
how to resolve conflicts with jar dependencies

i need to know how to resolve conflicts with jar dependencies.

* first, my job requires Jackson JSON-processor v1.9.11.
* second, the hadoop cluster has Jackson JSON-processor v1.5.2. the
jars are installed in $HADOOP_HOME/lib.

according to this link,
there are 3 ways to include 3rd party libraries in a map/reduce (mr)
* use the -libjars flag
* include the dependent libraries in the executing jar file's /lib directory
* put the jars in the $HADOOP_HOME/lib directory

i can report that using -libjars and including the libraries in my
jar's /lib directory "do not work" (in my case of jar conflicts). i
still get a NoSuchMethodException. the only way to get my job to run
is the last option, placing the newer jars in $HADOOP_HOME/lib. the
last option is fine on a sandbox or development instance, but there
are some political difficulties (not only technical) in modifying our
production environment.

my questions/concerns are:
1. how come the -libjars and /lib options do not work? how does class
loading work in mr tasks?
2. is there another option available that i am not aware of to try and
get dependent jars by the job to "overwrite" what's in
$HADOOP_HOME/lib at runtime of the mr tasks?
any help is appreciated. thank you all.