Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> how to resolve conflicts with jar dependencies


Copy link to this message
-
how to resolve conflicts with jar dependencies
hi,

i need to know how to resolve conflicts with jar dependencies.

* first, my job requires Jackson JSON-processor v1.9.11.
* second, the hadoop cluster has Jackson JSON-processor v1.5.2. the
jars are installed in $HADOOP_HOME/lib.

according to this link,
http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/,
there are 3 ways to include 3rd party libraries in a map/reduce (mr)
job.
* use the -libjars flag
* include the dependent libraries in the executing jar file's /lib directory
* put the jars in the $HADOOP_HOME/lib directory

i can report that using -libjars and including the libraries in my
jar's /lib directory "do not work" (in my case of jar conflicts). i
still get a NoSuchMethodException. the only way to get my job to run
is the last option, placing the newer jars in $HADOOP_HOME/lib. the
last option is fine on a sandbox or development instance, but there
are some political difficulties (not only technical) in modifying our
production environment.

my questions/concerns are:
1. how come the -libjars and /lib options do not work? how does class
loading work in mr tasks?
2. is there another option available that i am not aware of to try and
get dependent jars by the job to "overwrite" what's in
$HADOOP_HOME/lib at runtime of the mr tasks?
any help is appreciated. thank you all.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB