Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Java MapReduce Avro Jackson Error


Copy link to this message
-
Re: Java MapReduce Avro Jackson Error
If you are using avro-tools, beware it is a shaded jar with all dependencies
inside of it for use as a command line tool (java ­jar
avro-tools-VERSION.jar).

If you are using avro-tools in your project for some reason (there is really
only command line utilities in it) use the nodeps classifier:

<classifier>nodeps</classifier>

http://repo1.maven.org/maven2/org/apache/avro/avro-tools/1.6.3/

Note the nodeps jar is 47K, while the default jar is 10MB.
For what it is worth, I removed the Jackson jar from our hadoop install long
ago.  It is used to dump configuration files to JSON there, a peripheral
feature we don't use.

Another thing that you may want to do is change your Hadoop dependency scope
to
<scope>provided</scope> since hadoop will be put on your classpath by the
hadoop environment.   Short of this, excluding the chained Hadoop
dependencies you aren't using (most likely: jetty,  kfs, and the
tomcat:jasper and eclipse:jdt stuff) may help.

On 3/19/12 6:23 PM, "Deepak Nettem" <[EMAIL PROTECTED]> wrote:

> Hi Tatu / Scott,
>
> Thanks for your replies. I replaced the earlier dependencies with these:
>
>    <dependency>
>     <groupId>org.apache.avro</groupId>
>     <artifactId>avro-tools</artifactId>
>     <version>1.6.3</version>
>     </dependency>
>    
>     <dependency>
>     <groupId>org.apache.avro</groupId>
>     <artifactId>avro</artifactId>
>     <version>1.6.3</version>
>     </dependency>
>
>     <dependency>
>     <groupId>org.codehaus.jackson</groupId>
>       <artifactId>jackson-mapper-asl</artifactId>
>       <version>1.8.8</version>
>       <scope>compile</scope>
>     </dependency>
>
>     <dependency>
>     <groupId>org.codehaus.jackson</groupId>
>       <artifactId>jackson-core-asl</artifactId>
>       <version>1.8.8</version>
>       <scope>compile</scope>
>     </dependency>
>
> And this is my app dependency tree:
>
> [INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @ AvroTest ---
> [INFO] org.avrotest:AvroTest:jar:1.0-SNAPSHOT
> [INFO] +- junit:junit:jar:3.8.1:test (scope not updated to compile)
> [INFO] +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
> [INFO] +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
> [INFO] +- net.sf.json-lib:json-lib:jar:jdk15:2.3:compile
> [INFO] |  +- commons-beanutils:commons-beanutils:jar:1.8.0:compile
> [INFO] |  +- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO] |  +- commons-lang:commons-lang:jar:2.4:compile
> [INFO] |  +- commons-logging:commons-logging:jar:1.1.1:compile
> [INFO] |  \- net.sf.ezmorph:ezmorph:jar:1.0.6:compile
> [INFO] +- org.apache.avro:avro-tools:jar:1.6.3:compile
> [INFO] |  \- org.slf4j:slf4j-api:jar:1.6.4:compile
> [INFO] +- org.apache.avro:avro:jar:1.6.3:compile
> [INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
> [INFO] |  \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile
> [INFO] \- org.apache.hadoop:hadoop-core:jar:0.20.2:compile
> [INFO]    +- commons-cli:commons-cli:jar:1.2:compile
> [INFO]    +- xmlenc:xmlenc:jar:0.52:compile
> [INFO]    +- commons-httpclient:commons-httpclient:jar:3.0.1:compile
> [INFO]    +- commons-codec:commons-codec:jar:1.3:compile
> [INFO]    +- commons-net:commons-net:jar:1.4.1:compile
> [INFO]    +- org.mortbay.jetty:jetty:jar:6.1.14:compile
> [INFO]    +- org.mortbay.jetty:jetty-util:jar:6.1.14:compile
> [INFO]    +- tomcat:jasper-runtime:jar:5.5.12:compile
> [INFO]    +- tomcat:jasper-compiler:jar:5.5.12:compile
> [INFO]    +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:compile
> [INFO]    +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:compile
> [INFO]    |  \- ant:ant:jar:1.6.5:compile
> [INFO]    +- commons-el:commons-el:jar:1.0:compile
> [INFO]    +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
> [INFO]    +- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:compile
> [INFO]    +- net.sf.kosmosfs:kfs:jar:0.3:compile
> [INFO]    +- hsqldb:hsqldb:jar:1.8.0.10:compile
> [INFO]    +- oro:oro:jar:2.0.8:compile
> [INFO]    \- org.eclipse.jdt:core:jar:3.1.1:compile