Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Pig, mail # user - Problems loading a datafile..


Copy link to this message
-
Re: Problems loading a datafile..
Kris Coward 2011-02-28, 03:56

Oh, and I also get the same error if I omit the schema >:(

-K

On Sun, Feb 27, 2011 at 10:47:58PM -0500, Kris Coward wrote:
>
> So I finally got a couple of test scripts running on my cluster to take
> a sample data file, load it, do a little processing, store it, load it,
> do a little more processing, and dump the results.
>
> Once these were working, I set to parsing and storing some real data,
> but when got an "Unable to create input slice" error when trying to load
> this data back out again. This happened with each of:
>
> foo = LOAD '/path/to/file/{item,list,glob}/*/subdir' USING com.twitter.elephantbird.pig.load.LzoTokenizedLoader(',') AS (schema:...);
> foo = LOAD '/path/to/file/item/*/subdir' USING com.twitter.elephantbird.pig.load.LzoTokenizedLoader(',') AS (schema:...);
> foo = LOAD '/path/to/file/item/ex/subdir' USING com.twitter.elephantbird.pig.load.LzoTokenizedLoader(',') AS (schema:...);
>
> and yielded the error (the same each time, except for the name/glob
> used):
>
> ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Unable to create input slice for: hdfs://master.hadoop:9000//path/to/file/item/ex/subdir
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias foo
>         at org.apache.pig.PigServer.openIterator(PigServer.java:482)
>         at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
>         at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
>         at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
>         at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
>         at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:75)
>         at org.apache.pig.Main.main(Main.java:352)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Unable to create input slice for: hdfs://master.hadoop:9000/path/to/file/item/ex/subdir
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:176)
>         at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:253)
>         at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:249)
>         at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:781)
>         at org.apache.pig.PigServer.store(PigServer.java:529)
>         at org.apache.pig.PigServer.openIterator(PigServer.java:465)
>         ... 6 more
>
>
> Anyone have any suggestions why this may be happening and how to fix it?
>
> Thanks,
> Kris
>
> --
> Kris Coward http://unripe.melon.org/
> GPG Fingerprint: 2BF3 957D 310A FEEC 4733  830E 21A4 05C7 1FEB 12B3

--
Kris Coward http://unripe.melon.org/
GPG Fingerprint: 2BF3 957D 310A FEEC 4733  830E 21A4 05C7 1FEB 12B3