Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Plain View
Pig >> mail # user >> Support for Hadoop 2.2


+
Claudio Romo Otto 2013-10-24, 12:37
+
Prashant Kommireddi 2013-10-25, 00:33
+
Claudio Romo Otto 2013-10-25, 01:34
Copy link to this message
-
Re: Support for Hadoop 2.2
It seems that Pig was correctly compiled against Hadoop 23, but the Cassandra piece was not, check out the where the exception is coming from:

> Caused by: java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.JobContext, but class was expected
>     at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSplits(AbstractColumnFamilyInputFormat.java:113)

So, I would say that you also need to get Hadoop 2 compatible Cassandra connector first.

Jarcec

On Thu, Oct 24, 2013 at 10:34:49PM -0300, Claudio Romo Otto wrote:
> After change from hadoop20 to hadoop23 the warning dissapeared but I
> got the same exception (Found interface
> org.apache.hadoop.mapreduce.JobContext, but class was expected)
>
> I have tried over a fresh install: hadoop 2.2.0 and pig 0.12.1
> compiled by me, no other product nor configuration, just two
> servers, one master with ResourceManager and NameNode, one slave
> with DataNode and NodeManager.
>
> I can't understand why over this fresh cluster Pig 0.12 fails. Here
> is the new trace:
>
> 2013-10-24 16:10:52,351 [JobControl] ERROR
> org.apache.pig.backend.hadoop23.PigJobControl - Error while trying
> to run jobs.
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
>     at org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:130)
>     at org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
>     at java.lang.Thread.run(Thread.java:724)
>     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:257)
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:606)
>     at org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
>     ... 3 more
> Caused by: java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.JobContext, but class was expected
>     at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSplits(AbstractColumnFamilyInputFormat.java:113)
>     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
>     at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:491)
>     at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:508)
>     at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
>     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:415)
>     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>     at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
>     ... 8 more
>
>
> El 24/10/13 21:33, Prashant Kommireddi escribió:
> >Yes it does. You need to recompile Pig for hadoop 2
> >
> >ant clean jar-withouthadoop -Dhadoopversion=23
> >
> >
> >On Thu, Oct 24, 2013 at 5:37 AM, Claudio Romo Otto <
> >[EMAIL PROTECTED]> wrote:
> >
> >>Does Pig support Hadoop 2.2? When I try Pig 0.12 and Hadoop 2.2 I get an
> >>error even with simple operations like
> >>
> >>data = LOAD 'cql://keyspace1/testcf?' USING CqlStorage();
> >>dump data;
> >>
> >>I only got a warning first and then and exception:
> >>
> >>2013-10-24 09:35:19,300 [main] WARN org.apache.pig.backend.**hadoop20.PigJobControl
> >>- falling back to default JobControl (not using hadoop 0.20 ?)
> >>java.lang.**NoSuchFieldException: runnerState
> >>     at java.lang.Class.**getDeclaredField(Class.java:**1938)
+
Claudio Romo Otto 2013-10-25, 19:10
+
Jarek Jarcec Cecho 2013-10-30, 21:03
+
Claudio Romo Otto 2013-10-30, 22:40
+
Juan Martin Pampliega 2013-11-26, 19:07
+
Claudio Romo Otto 2013-11-26, 20:32
+
Uttam Kumar 2013-11-27, 00:54