Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
Avro >> mail # user >> Java MapReduce Avro Jackson Error


+
Deepak Nettem 2012-03-19, 22:48
+
Deepak Nettem 2012-03-19, 23:20
+
Tatu Saloranta 2012-03-19, 23:27
+
Deepak Nettem 2012-03-20, 00:05
+
Deepak Nettem 2012-03-20, 00:06
+
Scott Carey 2012-03-20, 01:06
+
Tatu Saloranta 2012-03-20, 01:12
+
Deepak Nettem 2012-03-20, 01:23
+
Tatu Saloranta 2012-03-20, 01:34
+
Something Something 2012-03-20, 01:43
Copy link to this message
-
Re: Java MapReduce Avro Jackson Error
If you are using avro-tools, beware it is a shaded jar with all dependencies
inside of it for use as a command line tool (java ­jar
avro-tools-VERSION.jar).

If you are using avro-tools in your project for some reason (there is really
only command line utilities in it) use the nodeps classifier:

<classifier>nodeps</classifier>

http://repo1.maven.org/maven2/org/apache/avro/avro-tools/1.6.3/

Note the nodeps jar is 47K, while the default jar is 10MB.
For what it is worth, I removed the Jackson jar from our hadoop install long
ago.  It is used to dump configuration files to JSON there, a peripheral
feature we don't use.

Another thing that you may want to do is change your Hadoop dependency scope
to
<scope>provided</scope> since hadoop will be put on your classpath by the
hadoop environment.   Short of this, excluding the chained Hadoop
dependencies you aren't using (most likely: jetty,  kfs, and the
tomcat:jasper and eclipse:jdt stuff) may help.

On 3/19/12 6:23 PM, "Deepak Nettem" <[EMAIL PROTECTED]> wrote:

> Hi Tatu / Scott,
>
> Thanks for your replies. I replaced the earlier dependencies with these:
>
>    <dependency>
>     <groupId>org.apache.avro</groupId>
>     <artifactId>avro-tools</artifactId>
>     <version>1.6.3</version>
>     </dependency>
>    
>     <dependency>
>     <groupId>org.apache.avro</groupId>
>     <artifactId>avro</artifactId>
>     <version>1.6.3</version>
>     </dependency>
>
>     <dependency>
>     <groupId>org.codehaus.jackson</groupId>
>       <artifactId>jackson-mapper-asl</artifactId>
>       <version>1.8.8</version>
>       <scope>compile</scope>
>     </dependency>
>
>     <dependency>
>     <groupId>org.codehaus.jackson</groupId>
>       <artifactId>jackson-core-asl</artifactId>
>       <version>1.8.8</version>
>       <scope>compile</scope>
>     </dependency>
>
> And this is my app dependency tree:
>
> [INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @ AvroTest ---
> [INFO] org.avrotest:AvroTest:jar:1.0-SNAPSHOT
> [INFO] +- junit:junit:jar:3.8.1:test (scope not updated to compile)
> [INFO] +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
> [INFO] +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
> [INFO] +- net.sf.json-lib:json-lib:jar:jdk15:2.3:compile
> [INFO] |  +- commons-beanutils:commons-beanutils:jar:1.8.0:compile
> [INFO] |  +- commons-collections:commons-collections:jar:3.2.1:compile
> [INFO] |  +- commons-lang:commons-lang:jar:2.4:compile
> [INFO] |  +- commons-logging:commons-logging:jar:1.1.1:compile
> [INFO] |  \- net.sf.ezmorph:ezmorph:jar:1.0.6:compile
> [INFO] +- org.apache.avro:avro-tools:jar:1.6.3:compile
> [INFO] |  \- org.slf4j:slf4j-api:jar:1.6.4:compile
> [INFO] +- org.apache.avro:avro:jar:1.6.3:compile
> [INFO] |  +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
> [INFO] |  \- org.xerial.snappy:snappy-java:jar:1.0.4.1:compile
> [INFO] \- org.apache.hadoop:hadoop-core:jar:0.20.2:compile
> [INFO]    +- commons-cli:commons-cli:jar:1.2:compile
> [INFO]    +- xmlenc:xmlenc:jar:0.52:compile
> [INFO]    +- commons-httpclient:commons-httpclient:jar:3.0.1:compile
> [INFO]    +- commons-codec:commons-codec:jar:1.3:compile
> [INFO]    +- commons-net:commons-net:jar:1.4.1:compile
> [INFO]    +- org.mortbay.jetty:jetty:jar:6.1.14:compile
> [INFO]    +- org.mortbay.jetty:jetty-util:jar:6.1.14:compile
> [INFO]    +- tomcat:jasper-runtime:jar:5.5.12:compile
> [INFO]    +- tomcat:jasper-compiler:jar:5.5.12:compile
> [INFO]    +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:compile
> [INFO]    +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:compile
> [INFO]    |  \- ant:ant:jar:1.6.5:compile
> [INFO]    +- commons-el:commons-el:jar:1.0:compile
> [INFO]    +- net.java.dev.jets3t:jets3t:jar:0.7.1:compile
> [INFO]    +- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:compile
> [INFO]    +- net.sf.kosmosfs:kfs:jar:0.3:compile
> [INFO]    +- hsqldb:hsqldb:jar:1.8.0.10:compile
> [INFO]    +- oro:oro:jar:2.0.8:compile
> [INFO]    \- org.eclipse.jdt:core:jar:3.1.1:compile
+
Deepak Nettem 2012-03-27, 02:58
+
Scott Carey 2012-03-27, 03:25
+
Deepak Nettem 2012-03-29, 14:28
+
Ken Krugler 2012-03-20, 02:51
+
Deepak Nettem 2012-03-20, 03:04
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB