I am writing Avro records in Ruby using the avro ruby gem in 1.8.7.  I have
problems with loading these files sometimes.  As a result, I am unable to
write large files that are readable.

The exception I get is below.  Anyone have an idea what this means?  It
looks like Avro is having trouble parsing the schema.  The avro files parse
in Ruby and Python, just not Pig.  Are there more rigorous checks in Java?

Pig Stack Trace
---------------
ERROR 2998: Unhandled internal error.
org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;

java.lang.NoSuchMethodError:
org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;
at org.apache.avro.Schema.<clinit>(Schema.java:82)
at
org.apache.pig.piggybank.storage.avro.AvroStorageUtils.<clinit>(AvroStorageUtils.java:49)
at
org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:163)
at
org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:144)
at
org.apache.pig.piggybank.storage.avro.AvroStorage.getSchema(AvroStorage.java:269)
at
org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150)
at
org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109)
at
org.apache.pig.newplan.logical.visitor.LineageFindRelVisitor.visit(LineageFindRelVisitor.java:100)
at org.apache.pig.newplan.logical.relational.LOLoad.accept(LOLoad.java:218)
at
org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
at
org.apache.pig.newplan.logical.visitor.CastLineageSetter.<init>(CastLineageSetter.java:57)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1679)
at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1610)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:495)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
===============================================================================
--
Russell Jurney
twitter.com/rjurney
[EMAIL PROTECTED]
datasyndrome.com

Spoken too soon... this happens no matter what avros I load now.  I can't
figure that anything has changed regarding jars, etc.  Confused.

I think this happens when Avro is parsing the schema?

Pig Stack Trace
---------------
ERROR 2998: Unhandled internal error.
org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;

java.lang.NoSuchMethodError:
org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;
at org.apache.avro.Schema.<clinit>(Schema.java:82)
at
org.apache.pig.piggybank.storage.avro.AvroStorageUtils.<clinit>(AvroStorageUtils.java:49)
at
org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:163)
at
org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:144)
at
org.apache.pig.piggybank.storage.avro.AvroStorage.getSchema(AvroStorage.java:269)
at
org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150)
at
org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109)
at
org.apache.pig.newplan.logical.visitor.LineageFindRelVisitor.visit(LineageFindRelVisitor.java:100)
at org.apache.pig.newplan.logical.relational.LOLoad.accept(LOLoad.java:218)
at
org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
at
org.apache.pig.newplan.logical.visitor.CastLineageSetter.<init>(CastLineageSetter.java:57)
at org.apache.pig.PigServer$Graph.compile(PigServer.java:1679)
at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1610)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:495)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
===============================================================================
On Thu, Feb 2, 2012 at 2:53 PM, Russell Jurney <[EMAIL PROTECTED]>wrote:

> Further examination shows that the problematic emails I am encoding are
> formatted in ISO-8859-1, not UTF-8.  That is why I am getting character
> problems.  Looks like it is not an Avro problem after all.  Thanks!  :)
>
>
> On Thu, Feb 2, 2012 at 2:49 PM, Russell Jurney <[EMAIL PROTECTED]>wrote:
>
>> A little bit more searching shows this:
>>
>>
>> http://www.harshj.com/2010/04/25/writing-and-reading-avro-data-files-using-python/
>>
>>
>> On Thu, Feb 2, 2012 at 2:48 PM, Russell Jurney <[EMAIL PROTECTED]>wrote:
>>
>>> The jars being used are:
>>>
>>> REGISTER /me/pig/build/ivy/lib/Pig/avro-1.5.3.jar
>>> REGISTER /me/pig/build/ivy/lib/Pig/json-simple-1.1.jar
>>> REGISTER /me/pig/contrib/piggybank/java/piggybank.jar
>>> REGISTER /me/pig/build/ivy/lib/Pig/jackson-core-asl-1.7.3.jar
>>> REGISTER /me/pig/build/ivy/lib/Pig/jackson-mapper-asl-1.7.3.jar
>>>
>>> On Thu, Feb 2, 2012 at 2:41 PM, James Baldassari <[EMAIL PROTECTED]>wrote:
>>>
>>>> HI Russell,
>>>>
>>>> I'm not sure about the Python error, but the Java error looks like a
>>>> classpath problem, not a schema parsing issue.  The NoSuchMethodError in
>>>> the stack trace indicates that Avro was trying to invoke a method in the
>>>> Jackson library that wasn't present at run-time.  My guess is that your

Russell Jurney
twitter.com/rjurney
[EMAIL PROTECTED]
datasyndrome.com

Cleaned up my environment by unsetting HADOOP_HOME, and removing some old
jacksons in my CLASSPATH and Pig's AvroStorage works again.

Woot!

On Thu, Feb 2, 2012 at 3:47 PM, Russell Jurney <[EMAIL PROTECTED]>wrote:

> Spoken too soon... this happens no matter what avros I load now.  I can't
> figure that anything has changed regarding jars, etc.  Confused.
>
> I think this happens when Avro is parsing the schema?
>
> Pig Stack Trace
> ---------------
> ERROR 2998: Unhandled internal error.
> org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;
>
> java.lang.NoSuchMethodError:
> org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;
> at org.apache.avro.Schema.<clinit>(Schema.java:82)
>  at
> org.apache.pig.piggybank.storage.avro.AvroStorageUtils.<clinit>(AvroStorageUtils.java:49)
> at
> org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:163)
>  at
> org.apache.pig.piggybank.storage.avro.AvroStorage.getAvroSchema(AvroStorage.java:144)
> at
> org.apache.pig.piggybank.storage.avro.AvroStorage.getSchema(AvroStorage.java:269)
>  at
> org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150)
> at
> org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109)
>  at
> org.apache.pig.newplan.logical.visitor.LineageFindRelVisitor.visit(LineageFindRelVisitor.java:100)
> at org.apache.pig.newplan.logical.relational.LOLoad.accept(LOLoad.java:218)
>  at
> org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
> at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
>  at
> org.apache.pig.newplan.logical.visitor.CastLineageSetter.<init>(CastLineageSetter.java:57)
> at org.apache.pig.PigServer$Graph.compile(PigServer.java:1679)
>  at org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1610)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582)
>  at org.apache.pig.PigServer.registerQuery(PigServer.java:584)
> at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942)
>  at
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386)
> at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188)
>  at
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
>  at org.apache.pig.Main.run(Main.java:495)
> at org.apache.pig.Main.main(Main.java:111)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
>  at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> ===============================================================================>
> On Thu, Feb 2, 2012 at 2:53 PM, Russell Jurney <[EMAIL PROTECTED]>wrote:
>
>> Further examination shows that the problematic emails I am encoding are
>> formatted in ISO-8859-1, not UTF-8.  That is why I am getting character
>> problems.  Looks like it is not an Avro problem after all.  Thanks!  :)
>>
>>
>> On Thu, Feb 2, 2012 at 2:49 PM, Russell Jurney <[EMAIL PROTECTED]>wrote:
>>
>>> A little bit more searching shows this:
>>>
>>>
>>> http://www.harshj.com/2010/04/25/writing-and-reading-avro-data-files-using-python/
>>>
>>>
>>> On Thu, Feb 2, 2012 at 2:48 PM, Russell Jurney <[EMAIL PROTECTED]
>>> > wrote:
>>>
>>>> The jars being used are:
>>>>
>>>> REGISTER /me/pig/build/ivy/lib/Pig/avro-1.5.3.jar
>>>> REGISTER /me/pig/build/ivy/lib/Pig/json-simple-1.1.jar
>>>> REGISTER /me/pig/contrib/piggybank/java/piggybank.jar
>>>> REGISTER /me/pig/build/ivy/lib/Pig/jackson-core-asl-1.7.3.jar
>>>> REGISTER /me/pig/build/ivy/lib/Pig/jackson-mapper-asl-1.7.3.jar

Russell Jurney
twitter.com/rjurney
[EMAIL PROTECTED]
datasyndrome.com