|
Milind Vaidya
2013-01-10, 19:40
Cheolsoo Park
2013-01-10, 19:49
Milind Vaidya
2013-01-10, 20:02
Cheolsoo Park
2013-01-10, 20:10
Milind Vaidya
2013-01-10, 20:18
Cheolsoo Park
2013-01-10, 20:21
Milind Vaidya
2013-01-10, 20:35
|
-
Pig and AveroStorage : nextMilind Vaidya 2013-01-10, 19:40
Avro Schema with int field
{ "type" : "record", "name" : "employee", "fields":[ {"name" : "name", "type" : "string", "default" : "NU"}, {"name" : "age", "type" : "int","default" : 0}, {"name" : "dept", "type": "string","default" : "DU"}, {"name" : "office", "type": "string","default" : "OU"}, {"name" : "salary", "type": "int", "default" : 0} ] } Avro Schema with float field { "type" : "record", "name" : "employee", "fields":[ {"name" : "name", "type" : "string", "default" : "NU"}, {"name" : "age", "type" : "int","default" : 0}, {"name" : "dept", "type": "string","default" : "DU"}, {"name" : "office", "type": "string","default" : "OU"}, {"name" : "salary", "type": "float","default" : 0.0} ] } I built the new piggbank.jar (pig 0.11) and used it for the 2 schemas containing int and float as above. Script: REGISTER /homes/immilind/HadoopLocal/Jars/avro-1.7.1.jar REGISTER /homes/immilind/HadoopLocal/Jars/piggybank.jar employee= load '/user/immilind/AvroData' using org.apache.pig.piggybank.storage.avro.AvroStorage(); dump employee; But I am getting following error : Backend error message --------------------- AttemptID:attempt_1357708865500_6931_m_000000_0 Info:Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected Backend error message --------------------- AttemptID:attempt_1357708865500_6931_m_000000_0 Info:Container killed by the ApplicationMaster. Backend error message --------------------- AttemptID:attempt_1357708865500_6931_m_000000_1 Info:Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected Backend error message --------------------- AttemptID:attempt_1357708865500_6931_m_000000_1 Info:Container killed by the ApplicationMaster. Backend error message --------------------- AttemptID:attempt_1357708865500_6931_m_000000_2 Info:Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected Backend error message --------------------- AttemptID:attempt_1357708865500_6931_m_000000_2 Info:Container killed by the ApplicationMaster. Backend error message --------------------- AttemptID:attempt_1357708865500_6931_m_000000_3 Info:Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected Pig Stack Trace --------------- ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1357708865500_6931_m_000000_3 Info:Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias employee. Backend error : Unable to recreate exception from backed error: AttemptID:attempt_1357708865500_6931_m_000000_3 Info:Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected at org.apache.pig.PigServer.openIterator(PigServer.java:826) at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320) at org.apache.pig.tools.grunt.GruntParser.loadScript(GruntParser.java:531) at org.apache.pig.tools.grunt.GruntParser.processScript(GruntParser.java:474) at org.apache.pig.tools.pigscript.parser.PigScriptParser.Script(PigScriptParser.java:804) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:449) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170) at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) at org.apache.pig.Main.run(Main.java:539) at org.apache.pig.Main.main(Main.java:158) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exception from backed error: AttemptID:attempt_1357708865500_6931_m_000000_3 Info:Error: Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:217) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:149) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:400) at org.apache.pig.PigServer.launchPlan(PigServer.java:1264) at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249) at org.apache.pig.PigServer.storeEx(PigServer.java:931) at org.apache.pig.PigServer.store(PigServer.java:898) at org.apache.pig.PigServer.openIterator(PigServer.java:811) ... 16 more ===============================================================================
-
Re: Pig and AveroStorage : nextCheolsoo Park 2013-01-10, 19:49
Hi Milind,
>> Found interface org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected This is an error when the Hadoop that you compiled Pig against doesn't match the Hadoop that you run Pig on. >> AttemptID:attempt_1357708865500_6931_m_000000_1 Info:Container killed by the ApplicationMaster. Looks like you're using Hadoop-2.0.x. If so, please recompile Pig with Hadoop-2.0.x: ant clean jar-withouthadoop.jar -Dhadoopversion=23 Thanks, Cheolsoo On Thu, Jan 10, 2013 at 11:40 AM, Milind Vaidya <[EMAIL PROTECTED]> wrote: > Avro Schema with int field > > { > "type" : "record", > "name" : "employee", > "fields":[ > {"name" : "name", "type" : "string", "default" : "NU"}, > {"name" : "age", "type" : "int","default" : 0}, > {"name" : "dept", "type": "string","default" : "DU"}, > {"name" : "office", "type": "string","default" : "OU"}, > {"name" : "salary", "type": "int", "default" : 0} > ] > } > > > Avro Schema with float field > { > "type" : "record", > "name" : "employee", > "fields":[ > {"name" : "name", "type" : "string", "default" : "NU"}, > {"name" : "age", "type" : "int","default" : 0}, > {"name" : "dept", "type": "string","default" : "DU"}, > {"name" : "office", "type": "string","default" : "OU"}, > {"name" : "salary", "type": "float","default" : 0.0} > ] > } > > I built the new piggbank.jar (pig 0.11) and used it for the 2 schemas > containing int and float as above. > > > Script: > > REGISTER /homes/immilind/HadoopLocal/Jars/avro-1.7.1.jar > REGISTER /homes/immilind/HadoopLocal/Jars/piggybank.jar > > employee= load '/user/immilind/AvroData' using > org.apache.pig.piggybank.storage.avro.AvroStorage(); > dump employee; > > But I am getting following error : > > Backend error message > --------------------- > AttemptID:attempt_1357708865500_6931_m_000000_0 Info:Error: Found interface > org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected > > Backend error message > --------------------- > AttemptID:attempt_1357708865500_6931_m_000000_0 Info:Container killed by > the ApplicationMaster. > > Backend error message > --------------------- > AttemptID:attempt_1357708865500_6931_m_000000_1 Info:Error: Found interface > org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected > > Backend error message > --------------------- > AttemptID:attempt_1357708865500_6931_m_000000_1 Info:Container killed by > the ApplicationMaster. > > Backend error message > --------------------- > AttemptID:attempt_1357708865500_6931_m_000000_2 Info:Error: Found interface > org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected > > Backend error message > --------------------- > AttemptID:attempt_1357708865500_6931_m_000000_2 Info:Container killed by > the ApplicationMaster. > > Backend error message > --------------------- > AttemptID:attempt_1357708865500_6931_m_000000_3 Info:Error: Found interface > org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected > > Pig Stack Trace > --------------- > ERROR 2997: Unable to recreate exception from backed error: > AttemptID:attempt_1357708865500_6931_m_000000_3 Info:Error: Found interface > org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to > open iterator for alias employee. Backend error : Unable to recreate > exception from backed error: > AttemptID:attempt_1357708865500_6931_m_000000_3 Info:Error: Found interface > org.apache.hadoop.mapreduce.TaskAttemptContext, but class was expected > at org.apache.pig.PigServer.openIterator(PigServer.java:826) > at > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696) > at > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320) > at > org.apache.pig.tools.grunt.GruntParser.loadScript(GruntParser.java:531) > at > org.apache.pig.tools.grunt.GruntParser.processScript(GruntParser.java:474)
-
Re: Pig and AveroStorage : nextMilind Vaidya 2013-01-10, 20:02
This gives me error as
Target "jar-withouthadoop.jar" does not exist in the project "pigudf". is it ok to use ant clean -Dhadoopversion=23 ? On Thu, Jan 10, 2013 at 11:49 AM, Cheolsoo Park <[EMAIL PROTECTED]>wrote: > jar-withouthadoop.jar
-
Re: Pig and AveroStorage : nextCheolsoo Park 2013-01-10, 20:10
Sorry. It's a typo.
ant clean jar-withouthadoop -Dhadoopversion=23 ant clean piggybank -Dhadoopversion=23 On Thu, Jan 10, 2013 at 12:02 PM, Milind Vaidya <[EMAIL PROTECTED]> wrote: > This gives me error as > > Target "jar-withouthadoop.jar" does not exist in the project "pigudf". > > is it ok to use > > ant clean -Dhadoopversion=23 ? > > > On Thu, Jan 10, 2013 at 11:49 AM, Cheolsoo Park <[EMAIL PROTECTED] > >wrote: > > > jar-withouthadoop.jar >
-
Re: Pig and AveroStorage : nextMilind Vaidya 2013-01-10, 20:18
Sorry same error.
( BTW ant clean jar-withouthadoop -Dhadoopversion=23 worked for me while compiling pig ) On Thu, Jan 10, 2013 at 12:10 PM, Cheolsoo Park <[EMAIL PROTECTED]>wrote: > Sorry. It's a typo. > > ant clean jar-withouthadoop -Dhadoopversion=23 > ant clean piggybank -Dhadoopversion=23 > > > On Thu, Jan 10, 2013 at 12:02 PM, Milind Vaidya <[EMAIL PROTECTED]> wrote: > > > This gives me error as > > > > Target "jar-withouthadoop.jar" does not exist in the project "pigudf". > > > > is it ok to use > > > > ant clean -Dhadoopversion=23 ? > > > > > > On Thu, Jan 10, 2013 at 11:49 AM, Cheolsoo Park <[EMAIL PROTECTED] > > >wrote: > > > > > jar-withouthadoop.jar > > >
-
Re: Pig and AveroStorage : nextCheolsoo Park 2013-01-10, 20:21
Sorry again for a typo.
ant clean jar-withouthadoop piggybank -Dhadoopversion=23 ant clean deleted pig-withouthadoop.jar and pig classes, so if you do ant clean piggybank followed by ant clean jar-withouthadoop, it will fail. On Thu, Jan 10, 2013 at 12:18 PM, Milind Vaidya <[EMAIL PROTECTED]> wrote: > Sorry same error. > > ( BTW ant clean jar-withouthadoop -Dhadoopversion=23 worked for me while > compiling pig ) > > > On Thu, Jan 10, 2013 at 12:10 PM, Cheolsoo Park <[EMAIL PROTECTED] > >wrote: > > > Sorry. It's a typo. > > > > ant clean jar-withouthadoop -Dhadoopversion=23 > > ant clean piggybank -Dhadoopversion=23 > > > > > > On Thu, Jan 10, 2013 at 12:02 PM, Milind Vaidya <[EMAIL PROTECTED]> > wrote: > > > > > This gives me error as > > > > > > Target "jar-withouthadoop.jar" does not exist in the project "pigudf". > > > > > > is it ok to use > > > > > > ant clean -Dhadoopversion=23 ? > > > > > > > > > On Thu, Jan 10, 2013 at 11:49 AM, Cheolsoo Park <[EMAIL PROTECTED] > > > >wrote: > > > > > > > jar-withouthadoop.jar > > > > > >
-
Re: Pig and AveroStorage : nextMilind Vaidya 2013-01-10, 20:35
Super....worked n loaded the data too...!
Thanks..! On Thu, Jan 10, 2013 at 12:21 PM, Cheolsoo Park <[EMAIL PROTECTED]>wrote: > Sorry again for a typo. > > ant clean jar-withouthadoop piggybank -Dhadoopversion=23 > > ant clean deleted pig-withouthadoop.jar and pig classes, so if you do ant > clean piggybank followed by ant clean jar-withouthadoop, it will fail. > > > On Thu, Jan 10, 2013 at 12:18 PM, Milind Vaidya <[EMAIL PROTECTED]> wrote: > > > Sorry same error. > > > > ( BTW ant clean jar-withouthadoop -Dhadoopversion=23 worked for me while > > compiling pig ) > > > > > > On Thu, Jan 10, 2013 at 12:10 PM, Cheolsoo Park <[EMAIL PROTECTED] > > >wrote: > > > > > Sorry. It's a typo. > > > > > > ant clean jar-withouthadoop -Dhadoopversion=23 > > > ant clean piggybank -Dhadoopversion=23 > > > > > > > > > On Thu, Jan 10, 2013 at 12:02 PM, Milind Vaidya <[EMAIL PROTECTED]> > > wrote: > > > > > > > This gives me error as > > > > > > > > Target "jar-withouthadoop.jar" does not exist in the project > "pigudf". > > > > > > > > is it ok to use > > > > > > > > ant clean -Dhadoopversion=23 ? > > > > > > > > > > > > On Thu, Jan 10, 2013 at 11:49 AM, Cheolsoo Park < > [EMAIL PROTECTED] > > > > >wrote: > > > > > > > > > jar-withouthadoop.jar > > > > > > > > > > |