Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> AvroStorage Issue - Possibly version related


+
j.barrett Strausser 2013-09-19, 16:23
+
Mark Wagner 2013-09-19, 18:24
+
j.barrett Strausser 2013-09-19, 19:14
+
j.barrett Strausser 2013-09-19, 20:10
+
Rohini Palaniswamy 2013-10-01, 00:54
Copy link to this message
-
Re: AvroStorage Issue - Possibly version related
I ended up just using the .23.9 hadoop release without any issue.
On Mon, Sep 30, 2013 at 8:54 PM, Rohini Palaniswamy <[EMAIL PROTECTED]
> wrote:

>  It hits this error when json-simple-1.1.jar is not in classpath. You can
> get around that by adding it to PIG_CLASSPATH apart from registering the
> jar.  The problem is with java classloading where it fails to load the
> exception class(ParseException) thrown by a constructor of the class(
> AvroStorage) from the custom classloader which includes the registered
> jars. Can you file a jira for this? I had spent some time earlier trying to
> find solution but couldn't. If fixing classloading is not possible, easy
> thing would be to change AvroStorage constructor to throw RunTimeException
> instead of ParseException
>
> Regards,
> Rohini
>
>
> On Thu, Sep 19, 2013 at 1:10 PM, j.barrett Strausser <
> [EMAIL PROTECTED]> wrote:
>
> > Not my day I guess.
> >
> > Trying with Hadoop 1.2.x
> >
> > Getting :
> >
> >
> > Caused by: java.lang.RuntimeException: could not instantiate
> > 'org.apache.pig.piggybank.storage.avro.AvroStorage' with arguments 'null'
> >         at
> >
> org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:618)
> >
> > Caused by: java.lang.NoClassDefFoundError:
> > org/json/simple/parser/ParseException
> >         at java.lang.Class.getDeclaredConstructors0(Native Method)
> >
> >
> >
> > When I attempt to load the relation using
> >
> > Load 'path' USING org.apache.pig.piggybank.storage.avro.AvroStorage();
> >
> > I've registered  :  json-simple-1.1.jar
> >
> >
> > On Thu, Sep 19, 2013 at 3:14 PM, j.barrett Strausser <
> > [EMAIL PROTECTED]> wrote:
> >
> > > Are the releases from the download page not compatible with 23.x? or
> 2.X
> > >
> > > Says they are -
> > >
> >
> http://pig.apache.org/releases.html#1+April%2C+2013%3A+release+0.11.1+available
> > >
> > > In any case I tried it with .23.9 and received a different error:
> > >
> > > 2013-09-19 15:13:56,044 [main] WARN
> > > org.apache.pig.backend.hadoop20.PigJobControl - falling back to default
> > > JobControl (not using hadoop 0.20 ?)
> > > java.lang.NoSuchFieldException: runnerState
> > >     at java.lang.Class.getDeclaredField(Class.java:1938)
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Sep 19, 2013 at 2:24 PM, Mark Wagner <[EMAIL PROTECTED]
> > >wrote:
> > >
> > >> It sounds like you're using a version of Pig that wasn't compiled for
> > >> Hadoop 2.x/.23. Try recompiling with 'ant clean jar
> > >> -Dhadoopversion=23'.
> > >>
> > >> -Mark
> > >>
> > >> On Thu, Sep 19, 2013 at 9:23 AM, j.barrett Strausser
> > >> <[EMAIL PROTECTED]> wrote:
> > >> > Running
> > >> >
> > >> > Hadoop-2.1.0-Beta
> > >> > Pig-0.11.1
> > >> > Hive-0.11.1
> > >> >
> > >> > 1. Created Avro backed table in Hive.
> > >> > 2. Loaded the table in Pig - records = Load '/path' USING
> > >> > org.apache.pig.piggybank.storage.avro.AvroStorage();
> > >> > 3. Can successfully describe the relation.
> > >> >
> > >> > I registered the following on pig start :
> > >> > REGISTER piggybank.jar
> > >> > REGISTER avro-*.jar
> > >> > REGISTER jackson-core-asl-1.8.8.jar
> > >> > REGISTER jackson-mapper-asl-1.8.8.jar
> > >> > REGISTER json-simple-1.1.jar
> > >> > REGISTER snappy-java-1.0.3.2.jar
> > >> >
> > >> > The avro tools are at 1.7.5
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > *
> > >> > *
> > >> > *Running Dump Produces the following*:
> > >> >
> > >> > 2013-09-19 12:08:21,639 [JobControl] ERROR
> > >> > org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error while
> > >> trying
> > >> > to run jobs.
> > >> > java.lang.IncompatibleClassChangeError: Found interface
> > >> > org.apache.hadoop.mapreduce.JobContext, but class was expected
> > >> >     at
> > >> >
> > >>
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
> > >> >     at
> > >> >
> > >>
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)

https://github.com/bearrito
@deepbearrito
+
Rohini Palaniswamy 2013-10-01, 14:00
+
j.barrett Strausser 2013-10-01, 14:37