Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Loading LZOs With Some JSON


Copy link to this message
-
Re: Loading LZOs With Some JSON
Eli Finkelshteyn 2011-09-12, 18:54
Hmmm, now it gets past my mention of the function, but when I run a dump
on generated information, I get:

2011-09-12 14:48:12,814 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 2997: Unable to recreate exception from backed error:
java.lang.ClassCastException: *org.apache.pig.data.DataByteArray cannot
be cast to java.lang.String*

Thanks for all the help so far!

Eli

On 9/12/11 2:42 PM, Dmitriy Ryaboy wrote:
> You also want json-simple-1.1.jar
>
>
> On Mon, Sep 12, 2011 at 10:46 AM, Eli Finkelshteyn<[EMAIL PROTECTED]>wrote:
>
>> Hmm, I'm loading up hadoop-lzo.*.jar, elephant-bird.*.jar, guava-*.jar, and
>> piggybank.jar, and then trying to use that UDF, but getting the following
>> error:
>>
>> ERROR 2998: Unhandled internal error. org/json/simple/parser/**
>> ParseException
>>
>> java.lang.**NoClassDefFoundError: org/json/simple/parser/**ParseException
>>         at java.lang.Class.forName0(**Native Method)
>>         at java.lang.Class.forName(Class.**java:247)
>>         at org.apache.pig.impl.**PigContext.resolveClassName(**
>> PigContext.java:426)
>>         at org.apache.pig.impl.**PigContext.**instantiateFuncFromSpec(**
>> PigContext.java:456)
>>         at org.apache.pig.impl.**PigContext.**instantiateFuncFromSpec(**
>> PigContext.java:508)
>>         at org.apache.pig.impl.**PigContext.**instantiateFuncFromAlias(**
>> PigContext.java:531)
>>         at org.apache.pig.impl.**logicalLayer.parser.**
>> QueryParser.EvalFuncSpec(**QueryParser.java:5462)
>>         at org.apache.pig.impl.**logicalLayer.parser.**
>> QueryParser.BaseEvalSpec(**QueryParser.java:5291)
>>         at org.apache.pig.impl.**logicalLayer.parser.**
>> QueryParser.UnaryExpr(**QueryParser.java:5187)
>>         at org.apache.pig.impl.**logicalLayer.parser.**
>> QueryParser.CastExpr(**QueryParser.java:5133)
>>         at org.apache.pig.impl.**logicalLayer.parser.**QueryParser.**
>> MultiplicativeExpr(**QueryParser.java:5042)
>>         at org.apache.pig.impl.**logicalLayer.parser.**
>> QueryParser.AdditiveExpr(**QueryParser.java:4968)
>>         at org.apache.pig.impl.**logicalLayer.parser.**
>> QueryParser.InfixExpr(**QueryParser.java:4934)
>>         at org.apache.pig.impl.**logicalLayer.parser.**QueryParser.**
>> FlattenedGenerateItem(**QueryParser.java:4861)
>>         at org.apache.pig.impl.**logicalLayer.parser.**QueryParser.**
>> FlattenedGenerateItemList(**QueryParser.java:4747)
>>         at org.apache.pig.impl.**logicalLayer.parser.**
>> QueryParser.GenerateStatement(**QueryParser.java:4704)
>>         at org.apache.pig.impl.**logicalLayer.parser.**
>> QueryParser.NestedBlock(**QueryParser.java:4030)
>>         at org.apache.pig.impl.**logicalLayer.parser.**
>> QueryParser.ForEachClause(**QueryParser.java:3433)
>>         at org.apache.pig.impl.**logicalLayer.parser.**
>> QueryParser.BaseExpr(**QueryParser.java:1464)
>>         at org.apache.pig.impl.**logicalLayer.parser.**
>> QueryParser.Expr(QueryParser.**java:1013)
>>         at org.apache.pig.impl.**logicalLayer.parser.**
>> QueryParser.Parse(QueryParser.**java:800)
>>         etc...
>>
>> Any ideas? I've verified that it recognizes the function itself, and that
>> the data it's running on is valid json. Not sure what else I can check.
>>
>> Eli
>>
>>
>>
>> On 9/9/11 7:13 PM, Dmitriy Ryaboy wrote:
>>
>>> They derive from the same classes as far as lzo handling goes, so I
>>> suspect
>>> something's up with your environment or inputs if you get
>>> LzoTokenizedLoader
>>> to work, but LzoJsonStorage does not.
>>>
>>> Note that LzoTokenizedLoader is deprecated -- just use LzoPigStorage.
>>>
>>> JsonLoader wouldn't work for you because it expects the complete input
>>> line
>>> to be json, not part of it. You want to load with LzoPigStorage, and then
>>> apply the JsonStringToMap udf to the third field.
>>>
>>> -D
>>>
>>>
>>> On Fri, Sep 9, 2011 at 3:49 PM, Eli Finkelshteyn<iefinkel@gmail.**com<[EMAIL PROTECTED]>>
>>>   wrote:
>>>
>>>   Hi,
>>>> I'm currently working on trying to load lzos that contain some JSON