Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - Maven dependency problems


Copy link to this message
-
Maven dependency problems
Matthias Friedrich 2012-08-04, 15:44
Hi,

I'm currently trying to fix Maven dependencies for Crunch and ran into
trouble with the POM for hadoop-core 1.0.3. It looks like the Maven
dependencies are different from the actual dependencies at runtime.

As a result, bugs caused by dependency conflicts won't show up until
runtime, makeing unit and integration tests in Maven useless for
detecting this class of problems. Unfortunately, these bugs tend to
be quite nasty.

For example, Hadoop ships a very old version of slf4j. If my job
includes a newer one this can lead to MethodNotFoundExceptions. Since
the dependency to slf4j is undeclared in hadoop-core, there is no way
to detect this using Maven dependency reports. Other conflicts may
lead to different behavior which is a lot harder to diagnose.

What's the best way to get this fixed? Or is there something I'm
missing?

Below I attached a report of the differences I found. I didn't list
the correct ones.

Thanks,
  Matthias

On the classpath as per org.apache.hadoop:hadoop-core:jar:1.0.3
---------------------------------------------------------------

ant-1.6.5.jar
commons-logging-1.0.3.jar
jackson-core-asl-1.0.1.jar
jackson-mapper-asl-1.0.1.jar
jets3t-0.7.1.jar
jsp-2.1-6.1.14.jar
jsp-api-2.1-6.1.14.jar
junit-3.8.1.jar
kfs-0.3.jar
servlet-api-2.5-6.1.14.jar
Actually on the classpath as per lib directory of hadoop-1.0.3-bin.tar.gz
-------------------------------------------------------------------------

asm-3.2.jar
aspectjrt-1.6.5.jar
aspectjtools-1.6.5.jar
commons-daemon-1.0.1.jar
commons-io-2.1.jar
commons-logging-1.1.1.jar
commons-logging-api-1.0.4.jar
hadoop-capacity-scheduler-1.0.3.jar
hadoop-fairscheduler-1.0.3.jar
hadoop-thriftfs-1.0.3.jar
jackson-core-asl-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jdeb-0.8.jar
jersey-core-1.8.jar
jersey-json-1.8.jar
jersey-server-1.8.jar
jets3t-0.6.1.jar
jsch-0.1.42.jar
junit-4.5.jar
kfs-0.2.2.jar
log4j-1.2.15.jar
mockito-all-1.8.5.jar
slf4j-api-1.4.3.jar
slf4j-log4j12-1.4.3.jar