Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Re: Pig JasonParser


Copy link to this message
-
Re: Pig JasonParser
Hi guys,

As for elephant-bird, it seems that it is not compatible with Pig 0.10
(CDH4) :(
I am using this configuration:
pig -version
Apache Pig version 0.10.0-cdh4.1.1 (rexported)
hadoop version
Hadoop 2.0.0-cdh4.1.1
and getting just the same error as Tim explained:
java.lang.IncompatibleClassChangeError: Found interface
org.apache.hadoop.mapreduce.Counter, but class was expected

I am running it with the following commands:
REGISTER elephant-bird-pig-3.0.2.jar;
inputData = LOAD 'sample_simple.json' USING
com.twitter.elephantbird.pig.load.JsonLoader() as (json:map[]);
DUMP inputData;
On Thu, Sep 27, 2012 at 8:48 AM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:

> Yep. It's just JsonLoader.
> By default it works on top of whatever's returned by TexInputFormat, but
> you can override that, as long as the input format returns a string that's
> valid json, we are cool (so in theory you could write a
> TwitterAPIInputFormat or something, and get the json in Pig, not that I
> would recommend that).
>
> D
>
> On Wed, Sep 26, 2012 at 9:34 PM, Russell Jurney <[EMAIL PROTECTED]
> >wrote:
>
> > Does that work without lzo?
> >
> > Russell Jurney http://datasyndrome.com
> >
> > On Sep 26, 2012, at 9:00 PM, Dmitriy Ryaboy <[EMAIL PROTECTED]> wrote:
> >
> > > Try asking Michael May on gihub? This seems to be an issue with his
> > Loader..
> > >
> > > The JsonLoader in ElephantBird should work in this case if you turn on
> > > nested parsing (
> > >
> >
> https://github.com/kevinweil/elephant-bird/blob/master/pig/src/main/java/com/twitter/elephantbird/pig/load/JsonLoader.java
> > > )
> > >
> > > D
> > >
> > > On Wed, Sep 26, 2012 at 2:31 PM, Deepak Tiwari <[EMAIL PROTECTED]>
> > wrote:
> > >
> > >> My bad.. I think I have compiled from
> > >> https://github.com/mmay/PigJsonLoader/blob/master/JsonLoader.javalong
> > >> time
> > >> back in my piggybank area..it indeed didnt come with the original
> jar...
> > >>
> > >> Regards,
> > >>
> > >> Deepak
> > >>
> > >> On Tue, Sep 25, 2012 at 8:14 AM, Bill Graham <[EMAIL PROTECTED]>
> > wrote:
> > >>
> > >>> I missed the part about Piggybank, but I'm confused because I don't
> see
> > >>> that class in SVN:
> > >>>
> > >>>
> > >>
> >
> http://svn.apache.org/viewvc/pig/branches/branch-0.10/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/
> > >>>
> > >>> Either way your error seems to be issues with parsing the doubles.
> > >>>
> > >>>
> > >>> On Mon, Sep 24, 2012 at 2:24 PM, Vivek Shrivastava <
> > >>> [EMAIL PROTECTED]
> > >>>> wrote:
> > >>>
> > >>>> Thanks for responding Bill, However I am using JsonLoader that is in
> > >> the
> > >>>> Piggybank with Pig-0.10.0.
> > >>>>
> > >>>> It doesnt need any schema and converts Json data as map (
> > >>>> org.apache.pig.piggybank.storage.JsonLoader() as (json:map[]) ) and
> I
> > >>>> extract data from there using keys. I have processed huge amount of
> > >> data
> > >>>> without any problem and no schema was required.
> > >>>>
> > >>>> Regards,
> > >>>>
> > >>>> Vivek
> > >>>>
> > >>>> On Mon, Sep 24, 2012 at 2:03 PM, Bill Graham <[EMAIL PROTECTED]>
> > >>> wrote:
> > >>>>
> > >>>>> This loader only works for data stored using JsonStorage. From the
> > >>>>> javadocs:
> > >>>>>
> > >>>>> A loader for data stored using
> > >>>>> JsonStorage<
> > >>>>>
> > >>>
> > >>
> >
> http://pig.apache.org/docs/r0.10.0/api/org/apache/pig/builtin/JsonStorage.html
> > >>>>>> .
> > >>>>>
> > >>>>> This is not a generic JSON loader. It depends on the schema being
> > >> stored
> > >>>>> with the data when conceivably you could write a loader that
> > >> determines
> > >>>>> the
> > >>>>> schema from the JSON.
> > >>>>>
> > >>>>> Was this data produced via JsonStorage? If not, you'll need to
> write
> > a
> > >>>>> custom loader.
> > >>>>>
> > >>>>> On Mon, Sep 24, 2012 at 12:04 PM, Deepak Tiwari <
> > [EMAIL PROTECTED]
> > >>>>>> wrote:
> > >>>>>
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> I am try to parse this data using Pig parser