Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Avro >> mail # user >> Re: Problem with Pig AvroStorage, with Avros that work in Ruby and Python


Copy link to this message
-
Re: Problem with Pig AvroStorage, with Avros that work in Ruby and Python
Further examination shows that the problematic emails I am encoding are
formatted in ISO-8859-1, not UTF-8.  That is why I am getting character
problems.  Looks like it is not an Avro problem after all.  Thanks!  :)

On Thu, Feb 2, 2012 at 2:49 PM, Russell Jurney <[EMAIL PROTECTED]>wrote:

> A little bit more searching shows this:
>
>
> http://www.harshj.com/2010/04/25/writing-and-reading-avro-data-files-using-python/
>
>
> On Thu, Feb 2, 2012 at 2:48 PM, Russell Jurney <[EMAIL PROTECTED]>wrote:
>
>> The jars being used are:
>>
>> REGISTER /me/pig/build/ivy/lib/Pig/avro-1.5.3.jar
>> REGISTER /me/pig/build/ivy/lib/Pig/json-simple-1.1.jar
>> REGISTER /me/pig/contrib/piggybank/java/piggybank.jar
>> REGISTER /me/pig/build/ivy/lib/Pig/jackson-core-asl-1.7.3.jar
>> REGISTER /me/pig/build/ivy/lib/Pig/jackson-mapper-asl-1.7.3.jar
>>
>> On Thu, Feb 2, 2012 at 2:41 PM, James Baldassari <[EMAIL PROTECTED]>wrote:
>>
>>> HI Russell,
>>>
>>> I'm not sure about the Python error, but the Java error looks like a
>>> classpath problem, not a schema parsing issue.  The NoSuchMethodError in
>>> the stack trace indicates that Avro was trying to invoke a method in the
>>> Jackson library that wasn't present at run-time.  My guess is that your
>>> program (or Pig?) either has two incompatible versions of the Jackson
>>> library on its classpath or maybe Avro's Jackson dependency has been
>>> excluded and a version that is incompatible with Avro is on the classpath.
>>>
>>> Which version of Avro is being used?  Running 'mvn dependency:tree' in
>>> Avro trunk I see that it's depending on Jackson 1.8.6.  Can you verify that
>>> only one version of Jackson is on the classpath and that it's the version
>>> that is required by whatever version of Avro is on the classpath?
>>>
>>> -James
>>>
>>>
>>>
>>> On Thu, Feb 2, 2012 at 5:21 PM, Russell Jurney <[EMAIL PROTECTED]
>>> > wrote:
>>>
>>>> Correction: when I read the file in Python, I get the error below.  It
>>>> looks like a unicode problem?  Can one tell Avro how to handle this?
>>>>
>>>> Traceback (most recent call last):
>>>>   File "./cat_avro", line 21, in <module>
>>>>     for record in df_reader:
>>>>   File
>>>> "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/avro-_AVRO_VERSION_-py2.6.egg/avro/datafile.py",
>>>> line 354, in next
>>>>     datum = self.datum_reader.read(self.datum_decoder)
>>>>   File
>>>> "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/avro-_AVRO_VERSION_-py2.6.egg/avro/io.py",
>>>> line 445, in read
>>>>     return self.read_data(self.writers_schema, self.readers_schema,
>>>> decoder)
>>>>   File
>>>> "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/avro-_AVRO_VERSION_-py2.6.egg/avro/io.py",
>>>> line 490, in read_data
>>>>     return self.read_record(writers_schema, readers_schema, decoder)
>>>>   File
>>>> "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/avro-_AVRO_VERSION_-py2.6.egg/avro/io.py",
>>>> line 690, in read_record
>>>>     field_val = self.read_data(field.type, readers_field.type, decoder)
>>>>   File
>>>> "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/avro-_AVRO_VERSION_-py2.6.egg/avro/io.py",
>>>> line 488, in read_data
>>>>     return self.read_union(writers_schema, readers_schema, decoder)
>>>>   File
>>>> "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/avro-_AVRO_VERSION_-py2.6.egg/avro/io.py",
>>>> line 654, in read_union
>>>>     return self.read_data(selected_writers_schema, readers_schema,
>>>> decoder)
>>>>   File
>>>> "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/avro-_AVRO_VERSION_-py2.6.egg/avro/io.py",
>>>> line 458, in read_data
>>>>     return self.read_data(writers_schema, s, decoder)
>>>>   File
>>>> "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/avro-_AVRO_VERSION_-py2.6.egg/avro/io.py",

Russell Jurney
twitter.com/rjurney
[EMAIL PROTECTED]
datasyndrome.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB