Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> AvroStorage Issue - Possibly version related


Copy link to this message
-
Re: AvroStorage Issue - Possibly version related
Thanks for the heads up and the solution.

I was trying to get pig working with avro in a prototype fashion so I
didn't have any version constraints.

On Tue, Oct 1, 2013 at 10:00 AM, Rohini Palaniswamy <[EMAIL PROTECTED]
> wrote:

> 0.23 has json-simple jar in the hadoop installation, so there is no
> problem. When you go to hadoop 2.x, you will hit the same problem as
> json-simple is not in 2.x hadoop installation. I actually hit the error you
> mentioned with 1.x while trying 2.x.
>
> Regards,
> Rohini
>
>
> On Mon, Sep 30, 2013 at 6:09 PM, j.barrett Strausser <
> [EMAIL PROTECTED]> wrote:
>
> > I ended up just using the .23.9 hadoop release without any issue.
> >
> >
> >
> >
> > On Mon, Sep 30, 2013 at 8:54 PM, Rohini Palaniswamy <
> > [EMAIL PROTECTED]
> > > wrote:
> >
> > >  It hits this error when json-simple-1.1.jar is not in classpath. You
> can
> > > get around that by adding it to PIG_CLASSPATH apart from registering
> the
> > > jar.  The problem is with java classloading where it fails to load the
> > > exception class(ParseException) thrown by a constructor of the class(
> > > AvroStorage) from the custom classloader which includes the registered
> > > jars. Can you file a jira for this? I had spent some time earlier
> trying
> > to
> > > find solution but couldn't. If fixing classloading is not possible,
> easy
> > > thing would be to change AvroStorage constructor to throw
> > RunTimeException
> > > instead of ParseException
> > >
> > > Regards,
> > > Rohini
> > >
> > >
> > > On Thu, Sep 19, 2013 at 1:10 PM, j.barrett Strausser <
> > > [EMAIL PROTECTED]> wrote:
> > >
> > > > Not my day I guess.
> > > >
> > > > Trying with Hadoop 1.2.x
> > > >
> > > > Getting :
> > > >
> > > >
> > > > Caused by: java.lang.RuntimeException: could not instantiate
> > > > 'org.apache.pig.piggybank.storage.avro.AvroStorage' with arguments
> > 'null'
> > > >         at
> > > >
> > >
> >
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:618)
> > > >
> > > > Caused by: java.lang.NoClassDefFoundError:
> > > > org/json/simple/parser/ParseException
> > > >         at java.lang.Class.getDeclaredConstructors0(Native Method)
> > > >
> > > >
> > > >
> > > > When I attempt to load the relation using
> > > >
> > > > Load 'path' USING
> org.apache.pig.piggybank.storage.avro.AvroStorage();
> > > >
> > > > I've registered  :  json-simple-1.1.jar
> > > >
> > > >
> > > > On Thu, Sep 19, 2013 at 3:14 PM, j.barrett Strausser <
> > > > [EMAIL PROTECTED]> wrote:
> > > >
> > > > > Are the releases from the download page not compatible with 23.x?
> or
> > > 2.X
> > > > >
> > > > > Says they are -
> > > > >
> > > >
> > >
> >
> http://pig.apache.org/releases.html#1+April%2C+2013%3A+release+0.11.1+available
> > > > >
> > > > > In any case I tried it with .23.9 and received a different error:
> > > > >
> > > > > 2013-09-19 15:13:56,044 [main] WARN
> > > > > org.apache.pig.backend.hadoop20.PigJobControl - falling back to
> > default
> > > > > JobControl (not using hadoop 0.20 ?)
> > > > > java.lang.NoSuchFieldException: runnerState
> > > > >     at java.lang.Class.getDeclaredField(Class.java:1938)
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Sep 19, 2013 at 2:24 PM, Mark Wagner <
> > [EMAIL PROTECTED]
> > > > >wrote:
> > > > >
> > > > >> It sounds like you're using a version of Pig that wasn't compiled
> > for
> > > > >> Hadoop 2.x/.23. Try recompiling with 'ant clean jar
> > > > >> -Dhadoopversion=23'.
> > > > >>
> > > > >> -Mark
> > > > >>
> > > > >> On Thu, Sep 19, 2013 at 9:23 AM, j.barrett Strausser
> > > > >> <[EMAIL PROTECTED]> wrote:
> > > > >> > Running
> > > > >> >
> > > > >> > Hadoop-2.1.0-Beta
> > > > >> > Pig-0.11.1
> > > > >> > Hive-0.11.1
> > > > >> >
> > > > >> > 1. Created Avro backed table in Hive.
> > > > >> > 2. Loaded the table in Pig - records = Load '/path' USING
> > > > >> > org.apache.pig.piggybank.storage.avro.AvroStorage();

https://github.com/bearrito
@deepbearrito