Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig >> mail # user >> LzoTokenizedStorage working, but now I can't get the data back out with LzoTokenizedLoader


Copy link to this message
-
LzoTokenizedStorage working, but now I can't get the data back out with LzoTokenizedLoader

After a bunch of fiddling around (including some pretty heavy use of the
secretDebugCmd--thanks), I finally got the LzoTokenizedStorage working,
but now I'm having problems with the LzoTokenizedLoader.

I'm still using pig 0.8.0-CDH3B4-SNAPSHOT, and for storage, have only
seemed to have luck with the hirohanin jarfile in hadoop-gpl; registered
all the other jarfiles from that package (i.e. google-collect-1.0.jar,
hadoop-lzo-0.4.8.jar, protobuf-java-2.3.0.jar, slf4j-api-1.5.8.jar,
slf4j-log4j12-1.5.10.jar, and yamlbeans-0.9.3.jar) and am now getting
the error ans stack trace:

ERROR 2997: Unable to recreate exception from backed error:
java.io.IOException: No codec for file
hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
not found, cannot run

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
to open iterator for alias test. Backend error : Unable to recreate
exception from backed error: java.io.IOException: No codec for file
hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
not found, cannot run
        at org.apache.pig.PigServer.openIterator(PigServer.java:742)
        at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
        at org.apache.pig.Main.run(Main.java:465)
        at org.apache.pig.Main.main(Main.java:107)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
2997: Unable to recreate exception from backed error:
java.io.IOException: No codec for file
hdfs://localhost/rawfiles/cf938c112909470baac12a1655e4d705/1295791200/16173945/pst/part-m-00000.lzo
not found, cannot run
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getErrorMessages(Launcher.java:221)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:151)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:337)
        at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:378)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1198)
        at org.apache.pig.PigServer.storeEx(PigServer.java:874)
        at org.apache.pig.PigServer.store(PigServer.java:816)
        at org.apache.pig.PigServer.openIterator(PigServer.java:728)
        ... 7 more

I'd think that with all the jarfiles I've loaded in (and the fact that
LzoTokenizedStorage worked fine), that there should be a codec kicking
around, so I'm kinda confused here..

Any help would be appreciated.

Thanks,
Kris

On Sun, Feb 13, 2011 at 03:00:13PM -0800, Dmitriy Ryaboy wrote:
> You are still getting an IncompatibleClassChangeError?
> That usually indicates there's something wrong with the classpath,
> like perhaps your version of Pig is not the one your UDFs (in this
> case, EB) was compiled against.
>
> Can you check your classpath? You can print out exactly what Pig is
> running by passing the "-secretDebugCmd" flag to bin/pig.
>
> D
>
> On Sun, Feb 13, 2011 at 11:00 AM, Kris Coward <[EMAIL PROTECTED]> wrote:
> >
> > Upgrading to pig 0.8.0-CDH3B4-SNAPSHOT, hasn't solved the problem, even
> > trying with each of of the 0.8.0 jar files (the hadoopgpl package used
> > provides 3 of them, one tagged as hirohanin's, another as gerritjvv's
> > and another as yours). I'm going to keep banging away at this, but would
> > be happy to hear any other ideas that might help.
> >
> > Thanks,
> > Kris
> >
> > On Fri, Feb 11, 2011 at 11:15:56AM -0800, Dmitriy Ryaboy wrote:
> >> The 0.7 branch of EB is not a real thing, it's hirohanin's first

Kris Coward http://unripe.melon.org/
GPG Fingerprint: 2BF3 957D 310A FEEC 4733  830E 21A4 05C7 1FEB 12B3