Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> Support for Hadoop 2.2


Copy link to this message
-
Re: Support for Hadoop 2.2
It seems that Pig was correctly compiled against Hadoop 23, but the Cassandra piece was not, check out the where the exception is coming from:

> Caused by: java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.JobContext, but class was expected
>     at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSplits(AbstractColumnFamilyInputFormat.java:113)

So, I would say that you also need to get Hadoop 2 compatible Cassandra connector first.

Jarcec

On Thu, Oct 24, 2013 at 10:34:49PM -0300, Claudio Romo Otto wrote:
> After change from hadoop20 to hadoop23 the warning dissapeared but I
> got the same exception (Found interface
> org.apache.hadoop.mapreduce.JobContext, but class was expected)
>
> I have tried over a fresh install: hadoop 2.2.0 and pig 0.12.1
> compiled by me, no other product nor configuration, just two
> servers, one master with ResourceManager and NameNode, one slave
> with DataNode and NodeManager.
>
> I can't understand why over this fresh cluster Pig 0.12 fails. Here
> is the new trace:
>
> 2013-10-24 16:10:52,351 [JobControl] ERROR
> org.apache.pig.backend.hadoop23.PigJobControl - Error while trying
> to run jobs.
> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
>     at org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:130)
>     at org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
>     at java.lang.Thread.run(Thread.java:724)
>     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:257)
> Caused by: java.lang.reflect.InvocationTargetException
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:606)
>     at org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
>     ... 3 more
> Caused by: java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.JobContext, but class was expected
>     at org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSplits(AbstractColumnFamilyInputFormat.java:113)
>     at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
>     at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:491)
>     at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:508)
>     at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
>     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:415)
>     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>     at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
>     ... 8 more
>
>
> El 24/10/13 21:33, Prashant Kommireddi escribió:
> >Yes it does. You need to recompile Pig for hadoop 2
> >
> >ant clean jar-withouthadoop -Dhadoopversion=23
> >
> >
> >On Thu, Oct 24, 2013 at 5:37 AM, Claudio Romo Otto <
> >[EMAIL PROTECTED]> wrote:
> >
> >>Does Pig support Hadoop 2.2? When I try Pig 0.12 and Hadoop 2.2 I get an
> >>error even with simple operations like
> >>
> >>data = LOAD 'cql://keyspace1/testcf?' USING CqlStorage();
> >>dump data;
> >>
> >>I only got a warning first and then and exception:
> >>
> >>2013-10-24 09:35:19,300 [main] WARN org.apache.pig.backend.**hadoop20.PigJobControl
> >>- falling back to default JobControl (not using hadoop 0.20 ?)
> >>java.lang.**NoSuchFieldException: runnerState
> >>     at java.lang.Class.**getDeclaredField(Class.java:**1938)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB