Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Support for Hadoop 2.2


Copy link to this message
-
Re: Support for Hadoop 2.2
Claudio Romo Otto 2013-10-25, 01:34
After change from hadoop20 to hadoop23 the warning dissapeared but I got
the same exception (Found interface
org.apache.hadoop.mapreduce.JobContext, but class was expected)

I have tried over a fresh install: hadoop 2.2.0 and pig 0.12.1 compiled
by me, no other product nor configuration, just two servers, one master
with ResourceManager and NameNode, one slave with DataNode and NodeManager.

I can't understand why over this fresh cluster Pig 0.12 fails. Here is
the new trace:

2013-10-24 16:10:52,351 [JobControl] ERROR
org.apache.pig.backend.hadoop23.PigJobControl - Error while trying to
run jobs.
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
     at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:130)
     at
org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
     at java.lang.Thread.run(Thread.java:724)
     at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:257)
Caused by: java.lang.reflect.InvocationTargetException
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
     at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:606)
     at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
     ... 3 more
Caused by: java.lang.IncompatibleClassChangeError: Found interface
org.apache.hadoop.mapreduce.JobContext, but class was expected
     at
org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat.getSplits(AbstractColumnFamilyInputFormat.java:113)
     at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
     at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:491)
     at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:508)
     at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:415)
     at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
     at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
     ... 8 more
El 24/10/13 21:33, Prashant Kommireddi escribi�:
> Yes it does. You need to recompile Pig for hadoop 2
>
> ant clean jar-withouthadoop -Dhadoopversion=23
>
>
> On Thu, Oct 24, 2013 at 5:37 AM, Claudio Romo Otto <
> [EMAIL PROTECTED]> wrote:
>
>> Does Pig support Hadoop 2.2? When I try Pig 0.12 and Hadoop 2.2 I get an
>> error even with simple operations like
>>
>> data = LOAD 'cql://keyspace1/testcf?' USING CqlStorage();
>> dump data;
>>
>> I only got a warning first and then and exception:
>>
>> 2013-10-24 09:35:19,300 [main] WARN org.apache.pig.backend.**hadoop20.PigJobControl
>> - falling back to default JobControl (not using hadoop 0.20 ?)
>> java.lang.**NoSuchFieldException: runnerState
>>      at java.lang.Class.**getDeclaredField(Class.java:**1938)
>>      at org.apache.pig.backend.**hadoop20.PigJobControl.<**
>> clinit>(PigJobControl.java:51)
>>      at org.apache.pig.backend.hadoop.**executionengine.shims.**
>> HadoopShims.newJobControl(**HadoopShims.java:97)
>>      at org.apache.pig.backend.hadoop.**executionengine.**mapReduceLayer.**
>> JobControlCompiler.compile(**JobControlCompiler.java:285)
>>      at org.apache.pig.backend.hadoop.**executionengine.**mapReduceLayer.**
>> MapReduceLauncher.launchPig(**MapReduceLauncher.java:177)
>>      at org.apache.pig.PigServer.**launchPlan(PigServer.java:**1264)
>>      at org.apache.pig.PigServer.**executeCompiledLogicalPlan(**